Enhancement suggestion: correlation id middleware combined with logger and sampler

[marcioemiranda]

Hello Heitor,

Have you considered adding a correlation_id middleware like the one in DAZN power tools?

It could work together with the logger tool by adding correlation_id to the logs automatically, and also be used by the sampling feature (the whole transaction is sampled instead of each function invoke).

Regards,
Marcio.

[heitorlessa]

Hey Marcio,

I did considered though I haven't been able to create a design document to flesh this out yet.

I'm currently working on another project until mid April, but if that's something you'd like to kick off I'd be super happy to contribute.

Thanks a lot for reaching out

[marcioemiranda]

Hello Heitor,

I would like to contribute if you give me some pointers.

Regards,
Marcio

[heitorlessa]

Absolutely @marcioemiranda - Here's a breakdown of how the logger works

• logger_setup returns a Python Logger with minimal structured logging. It also accepts any key=value argument that will be present across all log operations
• logger_inject_lambda_context is the decorator that intercepts the Lambda context object, and uses logger_setup to add key Lambda fn info as part of the struct log

I see two ways of implementing it:

• A) Extending logger_inject_lambda_context to detect whether there's a correlation ID already (event object)
• B) Create a separate module to both detect and hold that correlation ID in memory, so other future modules could make use of it (middleware kinda approach)

I personally prefer B as it's more extensible since we could also use the correlation ID data in Tracer to create an annotation, and in future modules like detecting retries, etc.

Regardless of the option, it'll have to add a new key correlation_id in this line: https://github.com/awslabs/aws-lambda-powertools/blob/develop/python/aws_lambda_powertools/logging/logger.py#L149

Happy to explain further but hopefully that makes sense

[heitorlessa]

I've decided to look at this after GA - Logger can append additional keys, and custom middlewares can be created now.

I want to spend more time in the design phase to make this scalable to multiple event sources too. I'll ping here when I start working on a design to get your ideas too.

[keithrozario]

This would be an awesome feature, and would be super useful for many folks.

[patrickwerz]

yes, definitely interested.

[heitorlessa]

Couldn't spend much quality time to think about the developer experience yet -- If anyone has the bandwidth, I'd appreciate a RFC for further discussion here.

[michaelbrewer]

Yes, this would be super useful. I will see if i have the headspace for this :)

[michaelbrewer]

I created a demo PR here:

• feat(logging): Add correlation_id support #321

[michaelbrewer]

FYI @marcioemiranda @patrickwerz @keithrozario the PR was merged for the first part of this.