RFC: Lambda Powertools for Python v2

[rubenfonseca]

Is this related to an existing feature request or issue?

No response

Which AWS Lambda Powertools utility does this relate to?

Powertools itself

Summary

The Python 3.6 Lambda Runtime was deprecated as of Aug 17th 2022. This follows Python 3.6 End-Of-Life (EOL) reached on December 23, 2021. This means it's not possible to create or update Lambda functions using that runtime.

We should take this opportunity to release a new major version of Powertools, and decide what breaking changes to include.

Use case

Customers need to upgrade their Lambda Functions to one of the supported Python runtimes (at the moment of writing, Python 3.7-3.9).

We could use this major upgrade to introduce some breaking changes on Lambda Powertools that will simplify the code and/or improve support for existing features.

Proposal

We strive to make minimal breaking changes due to timing constraints with Python 3.6 deprecation. In ideal scenarios, we would provide ample time, nightly builds, a Beta for a major version, including a linter to help detect and upgrade from breaking changes.

This means, V2 will focus primarily on the following themes:

• Drop Python 3.6. Follows alignment with Lambda runtime deprecation policy. It also allows us to upgrade development dependencies and documentation niceties we couldn't before due to prolonged Python 3.6 EOL life in Lambda.
• Make all dependencies optional to optimize package size. AWS SDK makes up for over 90% of our package size. We also heard from customers that they want to use additional observability providers, thus wanting X-Ray SDK to be optional.
  • Lambda Layer will include all optional dependencies excluding those already available at Lambda runtime to further optimize cold start. This also means dropping SAR Extras as it becomes redundant.
• Remove deprecated features. Specifically sqs_batch_processor and PartialSQSProcessor in favour of BatchProcessor launched 11 months ago that improved accuracy, security, and speed due to the new Lambda integration.
• Improve correctness. Event Handler (API Gateway) doesn't support multi-header and cookies by default as it requires a breaking change. Similarly, Idempotency and Tracer doesn't use fully qualified names, impacting ABC or Classes that use the exact same method name - changing it would impact billions of transactions in production.

Quick summary

Item	Issue/PR	Status	Code change required
Write What's new for v2
Update upgrade guide	#1623	✅
Remove POWERTOOLS_EVENT_HANDLER_DEBUG env var	#1620	✅
Drop support for python 3.6	#1339	✅
Remove the old batch processing legacy implementation (sqs_batch_processor, PartialSQSProcessor)	#1462	✅	Yes
Event Handler REST - multi-value Headers by default, and cookies support	#1455	✅	tests only
Use fully qualified names for idempotency	#1330	✅
Use fully qualified names for tracer subsegments	#1454	✅
Update AppConfig API in Parameters/Feature flags due to GetConfiguration deprecation	#1506	✅
Deprecate SAR Extras	#1543	✅
Make all runtime dependencies optional	#1164	✅
Update docs on required dependencies (validation, parser)	#1573	✅
Event Handler REST v1 supports trailing slash route by default	#1609	✅
Replace AttributeValue in DynamoDBStreamEvent with deserialized Python values	#1619	✅	Yes
Replace email-validator dependency with str in Parser SES Model	#1608	✅

Launch plan

• un-comment "deploy-prod" from publish_v2_layer.yml
• update docs to include SAR v2 and Layer v2
• review Upgrade Guide
• review What's New
• remove V2 banner from docs
• remove V2 admonition from docs

Out of scope

We should make the breaking release as small as possible. For this we should not include in v2:

• modularization to further reduce sizes
• Add a default unit for the metrics utility Provide Default Values for .add_metric(unit=) and .add_metric(value=) #1180

Potential challenges

We need to decide what to do with the Lambda Powertools Layer. One option would be to release a new "V2" layer.

UPDATE: We're launching a separate V2 layer and ARM support along with it!

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Thank you @rubenfonseca! Let's make Event Handler Headers part of v2 as this will impact our ability to support cookies and multi-value headers without a maintenance hit -- #1455

We should also add that we're going to make minimal breaking changes due to not having ample time to discuss and run experiments with customers - we shall leave those to 3.0 (modularization, potentially making pydantic the default, etc.)

[rubenfonseca]

@heitorlessa thanks, updated! Do you know if an existing issue about removing the first batch processing implementation?

[heitorlessa]

Nope, haven't created yet. We'd need to make a PR to add a deprecation warning too.

https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/aws_lambda_powertools/utilities/batch/sqs.py#L24

[rubenfonseca]

We started creating an "Upgrade guide" with detailed upgrade steps for each breaking change, together with before/after examples. This will help everyone go faster through each major version changes.

[heitorlessa]

I'd like to propose we make all dependencies optional until we modularise in V3.

Although we managed to reduce >90% of AWS SDK, keeping AWS X-Ray SDK as a main dependency brings the full AWS SDK - this won't benefit customers using pip install.

This would be a boost for customers not using X-Ray, those with sensitive startup time, and reduce build/deployment times.

Compressed

 11M	lambda_powertools_v1.zip
464K	lambda_powertools_v2.zip

Uncompressed

 75M	/tmp/lambda_powertools_v1
1.8M	/tmp/lambda_powertools_v2

[heitorlessa]

Early days but we started documenting the breaking changes in v2: https://awslabs.github.io/aws-lambda-powertools-python/v2/upgrade/

@rubenfonseca one thing we missed is adding another bullet point on multiValueHeaders for REST v1 payload

[mew1033]

I'd love to see strict typing supported in v2. As an example: #1089

[rubenfonseca]

Added #1506 to v2

[heitorlessa]

@kapilt and everyone else interested in smaller package size - @rubenfonseca just made a breakthrough for V2.

Our new Lambda Layer (v2) will be just 1M 2M with all dependencies installed (using AWS SDK available at Lambda runtime, of course).

For V2 launch this month, @rubenfonseca is also completing the work for an ARM Lambda Layer.

[heitorlessa]

Last update before we launch 2.0 this week

release manager: @rubenfonseca

We're working on one last change:

• Event Source Data Classes DynamoDB Stream Event. Return deserialized Dict for new_image and old_image instead of AttributeValue. This will unblock customers accessing nested map objects, simplify access to DynamoDB record data, and near-interoperability with Boto3 TypeDeserializer (exception being str instead of Decimal). [PR]

In parallel, we're updating (1) GitHub Actions workflows to support v1 and v2 releases, (2) Upgrade Guide to ensure it is as clear as it can be, (3) clean up this RFC body to reflect where we are and share V1 will be in maintenance mode until Jan 31st.

Until end of the week

You should expect a (A) lengthy Release Notes (What's New in 2.0), a (B) detailed Upgrade Guide for three breaking changes that might affect you, (C) a significant package size reduction (10M-->204k), and a (D) much improved Lambda Layer with all dependencies optimized (14M-->2M), including ARM support!

---

@mew1033 @peterschutt after V2 is launched, let's bring typing_extensions as a runtime dependency to start working on strict typing for the entire code base. We've made significant efforts in reducing package size (>95%), and bringing typing_extensions outweigh the effort of various workarounds to make typing work for multiple Python versions.

This will open a future door that we could use mypyc to compile certain parts of our code base to be a C-extension for extra speed. Now that we fully migrated our Lambda Layer pipeline and already use Cython for Parser, it'll be a no brainier to generate platform wheels too when the time is appropriate.

[peterschutt]

@mew1033 @peterschutt after V2 is launched, let's bring typing_extensions as a runtime dependency to start working on strict typing for the entire code base. We've made significant efforts in reducing package size (>95%), and bringing typing_extensions outweigh the effort of various workarounds to make typing work for multiple Python versions.

Sounds great! I'll touch base with you after the v2 launch, congrats!

[bryantbiggs]

ya'll are awesome - great work @heitorlessa / @rubenfonseca 🎉

[heitorlessa]

All Ready! We're gonna publish a pre-release to test all workflows, and then we start writing the official Release Notes for 2.0.

Finally! 🎉

[rubenfonseca]

We just released AWS Lambda Powertools Python v2! 🎉🎉🎉🎉

Don't forget to check our Upgrade Guide too :)

Thank you everyone for your help getting this our of the door! 🙏 Onwards and upwards! 🎉

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.