Access to the attributes and message_attributes of the underlying SQS message

[gvaughn]

It appears that in wrap_messages only the body from the message ExAws generates here is kept. I have a situation with an encrypted body that delivers the initialization vector needed for decryption in a MessageAttribute. Is there some other way of accessing this information?

[gvaughn]

Combined with this and #11 it is looking more like my needs correspond with a custom SQSClient rather than adding complexity to the core of Broadway.

[msaraiva]

Hi @gvaughn!

I think we should allow BroadwaySQS to expose messages attributes. The big question is how.

First, we need to add a field to the Message struct to hold that information. It could be attributes, in case the concept can be extended to other connectors, or something like extra_data, which is probably better since we're not assuming any contract in advance.

Another thing to consider is whether we should always add all existing attributes to the message or if we should provide an option so the user could list only the attributes he needs.

What do you think?

[josevalim]

I like a field called metadata or attributes and explicitly requesting the user to list which ones should be extracted. -- *José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D

[gvaughn]

I've been taking some time with a custom SQSClient to better understand my overall challenges. I learned that I have to pass options into ExAwsSQS.receive_messages to opt into the attributes and message_attributes (which seem to be different for a reason I don't understand yet). Those won't be passed through via ExAwsClient.validate_receive_message_opts in the existing code either.

The service provider I'm integrating with seems to be quite outside the norm for SQS activity in general. At this point I think it's best for me to push forward with the custom SQSClient to get things working and then from that seeing what extra flexibility should be put into BroadwaySQS after I have a working implementation.

Thanks for the feedback!

[pedromtavares]

Accessing message attributes via Broadway would be fantastic. I've been refactoring a custom data pipeline to use Broadway but got stuck in this part since we use message attributes to structure how our data gets passed around as we pass multiple keys through each message. Completely agree with @josevalim's idea of requesting the user to list which attributes would be needed.

[msaraiva]

I learned that I have to pass options into ExAwsSQS.receive_messages to opt into the attributes and message_attributes (which seem to be different for a reason I don't understand yet). Those won't be passed through via ExAwsClient.validate_receive_message_opts in the existing code either.

I'll take a closer look at SQS API regarding message attributes so we can propose a general solution.

At this point I think it's best for me to push forward with the custom SQSClient to get things working and then from that seeing what extra flexibility should be put into BroadwaySQS after I have a working implementation.

Sounds like a good plan. I created an issue for Broadway to add the attributes field since any solution will depend on it: dashbitco/broadway#69