Nested events and embedded deserialisable objects

[sdd]

Quite frequently, I've got a use case where I have a Lambda subscribed to an SQS Queue that is in turn subscribed to an SNS topic, to which I'm posting a custom struct serialized to JSON as the message.

It would be great to have some way of having SQS and SNS events that, instead of having message / body attributes that are String, are able to JSON decode their payload so that I can for example define functions like this:

#[derive(Deserialize)]
struct CustomMsgStruct {
  // ... blah
}

pub async fn handler(e: LambdaEvent<SqsEvent<SnsEvent<CustomMsgStruct>>>) -> Result<(), Error> {
    
    let sqs_messages: Vec<SnsEvent<CustomMsgStruct>> = e.payload;

    for message in sqs_messages.into_iter() {
        let sns_records: Vec<SnsRecord<CustomMsgStruct>> = message.body.records;

        for record in sns_records.into_iter() {
            let msg: CustomMsgStruct = record.sns.message;
        
            // ... Do whatever I need to do with my deserialized custom struct
        }
    }

    Ok(())
}

Are there any good patterns out there that people have come across for doing this ergonomically? I'm currently using the SqsEvent from the lambda_sqs crate to handle deserializing SnsEvents out of the SQS message, followed by just calling serde_json manually on the SnsMessage body, but it feels very clunky to do this and it feels like something that it would be nice for aws_lambda_events to support itself.

[sdd]

I might throw something together on my fork using https://docs.rs/serde_with/latest/serde_with/json/nested/index.html and comment back here on how well it works.

[calavera]

See what I did with the cloudwatch logs, it might help you:

https://github.com/LegNeato/aws-lambda-events/blob/master/aws_lambda_events/src/cloudwatch_logs/mod.rs#L52

[sdd]

Thanks. Once concern I have though is for users that aren't embedding JSON and just want the original string, something like SqsMessage<String> won't work as expected since the nested JSON decoder will expect to see an incoming event with nested quotes in order to deserialize that correctly: i.e., { ... body: "\"message\"" .. } instead of { ... body: "message" .. }.

A trivial solution would be to leave SqsEvent, SqsMessage, SnsMessage, SnsRecord, and SnsEvent as they are now so that the non-embedded-JSON use case works, and introduce SqsMessageObj<T>, SnsMessageObj<T>, etc structs.

I'm not sure what approach to take that could allow both approaches to co-exist by altering the existing structs, rather than introducing these parallel generic ones. Any suggestions?

[sdd]

I've got this working on a branch on my fork using the simple but slightly inelegant solution outlined above.

Now I can do this:

pub async fn handler(e: LambdaEvent<SqsEventObj<SnsMessageObj<SolveResponse>>>, app_ctx: &AppContext) -> Result<(), anyhow::Error> {

    for sqs_msg in e.payload.records.into_iter() {
        msg_handler(sqs_msg.body.message, app_ctx).await?;
    }

    Ok(())
}

instead of this:

use lambda_sqs::{Error, SqsEvent};

pub async fn handler(e: LambdaEvent<SqsEvent>, app_ctx: &AppContext) -> Result<(), Error> {
    let messages: Vec<SnsMessage> = e.payload.into_t(); // from lambda_sqs

    for message in messages.into_iter() {
        let msg = serde_json::from_str(&message.Message)?;
        msg_handler(msg, app_ctx).await?;
    }

    Ok(())
}

[calavera]

A trivial solution would be to leave SqsEvent, SqsMessage, SnsMessage, SnsRecord, and SnsEvent as they are now so that the non-embedded-JSON use case works, and introduce SqsMessageObj<T>, SnsMessageObj<T>, etc structs.

I think this is a good approach.