Feature request: Start Evaluating Pydantic v2

[djfurman]

Use case

Pydantic is already included in aws-lamdba-powertools as the provider for deep data validation and parsing as an optional dependency. Pydantic V2 is now in beta evaluation, when it becomes a stable version, it should be introduced as an option into powertools without major breaking changes. We need a way to test the integration with the rest of the library and benchmark performance within Lambda.

Solution/User Experience

Today the parser is included as an extra using Pydantic. With the launch of PydanticV2 we could include a second extra, add max version, or identify if we can do this transparently to the user with an optional switch.

To keep powertools lean, having both dependencies on a single extra seems less than ideal.

What is the best way to evaluate an impact like this without falling out of sync with the enhancements to powertools?

Alternative solutions

- Have two extras tagged to Pydantic versions
- Freeze the parser to Pydantic v1
- Identify if there are any breaking implementations within parser for breaking API changes of the Pydantic upgrade (not personally recommended)
- Identify how to handle the version difference in the full install layer as users will choose one version, not both in their function.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

### Tasks

[djfurman]

Pydantic does not provide a Java or .NET library, nor am I aware of one for TypeScript.

[heitorlessa]

Hey Daniel, good idea!! Have they settled on major breaking changes for Pydantic V2? If not, any major ones that stands out already? We knew v2 was in the works back then so we kept our usage to the bare minimum hoping the transition would be seamless. I suspect the only one surprise might be the coercion mode they changed now (strict vs coerce). Happy with a new optional dep for the time being until we figure this out. Layer will be tricky, it’d require a major version unless the transition is painless — we’d need plenty of guidance like our upgrade guide either way. Thanks for kicking the tires ;)

…

On Thu, 8 Jun 2023 at 20:59, Daniel Furman ***@***.***> wrote: Pydantic does not provide a Java or .NET library, nor am I aware of one for TypeScript. — Reply to this email directly, view it on GitHub <#2427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBFUP75VIHVMDNFPXCLXKIOIJANCNFSM6AAAAAAY7V5Q2M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[half2me]

Pydantic 2 has now been released

[leandrodamascena]

Hey everyone! By this month's end, we will create an RFC to discuss the migration from Pydantic v1 to v2. In addition to the breaking changes this migration will cause, we want to understand if there is a way to keep both versions of Pydantic in Powertools and can let customers choose which one they want to use.

Once we have the RFC, I will update this issue.

[half2me]

@leandrodamascena I use both powertools and pydantic in my lambda. Is there some way for me to use pydantic 2 while also letting powertools continue to use v1? If I try to update it, it complains that powertools is pinning the older version

[leandrodamascena]

@leandrodamascena I use both powertools and pydantic in my lambda. Is there some way for me to use pydantic 2 while also letting powertools continue to use v1? If I try to update it, it complains that powertools is pinning the older version

I don't think you can @half2me! Because we still don't support Pydantic v2 and if you try to use Powertools with this version you will get errors. We are prioritizing efforts to support Pydantic v2 as soon as possible.

[leandrodamascena]

Hey everyone! I started working with Pydantic v2 and Powertools and I could use SqsModel with Pydantic v2. I need to dig deeper and investigate this migration further, especially as we want to support v1 and v2 at the same time to avoid breaking changes.

Things I've discovered so far:

1 - All optional fields must have a default value of None otherwise Pydantic will fail.

Pydantic v1

class SqsAttributesModel(BaseModel):
    ApproximateReceiveCount: str
    ApproximateFirstReceiveTimestamp: datetime
    MessageDeduplicationId: Optional[str]
    MessageGroupId: Optional[str]
    SenderId: str
    SentTimestamp: datetime
    SequenceNumber: Optional[str]
    AWSTraceHeader: Optional[str]

Pydantic v2

class SqsAttributesModel(BaseModel):
    ApproximateReceiveCount: str
    ApproximateFirstReceiveTimestamp: datetime
    MessageDeduplicationId: Optional[str] = None
    MessageGroupId: Optional[str] = None
    SenderId: str
    SentTimestamp: datetime
    SequenceNumber: Optional[str] = None
    AWSTraceHeader: Optional[str] = None

2 - The @root_validator decorator is deprecated and will be removed in v3. Even though Pydantic suggests using the new @model_validator decorator, we can still use the @root_validator decorator to avoid breaking changes and plan this removal for Powertools v3.

Pydantic v1

@root_validator(allow_reuse=True)
def check_message_id(cls, values):
    message_id, event_type = values.get("messageId"), values.get("eventType")
    if message_id is not None and event_type != "MESSAGE":
        raise TypeError("messageId is available only when the `eventType` is `MESSAGE`")
    return values

Pydantic v2

@root_validator(allow_reuse=True, skip_on_failure=True)
def check_message_id(cls, values):
    message_id, event_type = values.get("messageId"), values.get("eventType")
    if message_id is not None and event_type != "MESSAGE":
        raise TypeError("messageId is available only when the `eventType` is `MESSAGE`")
    return values

3 - Pydantic package size dropped from ~9MB to ~5.5MB

4 - Let's take this opportunity to check if we can leverage this new version and make some changes to improve the performance in the use of Pydantic.

---

Exampled used:

from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, envelopes
from aws_lambda_powertools.utilities.parser.models import (
    SqsModel,
)

from aws_lambda_powertools import Logger
import pydantic

logger = Logger()

@event_parser(model=SqsModel)
def lambda_handler(event: SqsModel, context):
    logger.info(f"Pydantic version -> {pydantic.__version__}")
    for record in event.Records:
        logger.info(f"Event body -> {record.body}")

Output

❯ sam local invoke --event events/sqs.json
Invoking app.lambda_handler (python3.10)
Local image is up-to-date
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.

Mounting /home/leandro/DEVEL-PYTHON/tmp/pydantic2/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631 Version: $LATEST
START RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631 Version: $LATEST
{"level":"INFO","location":"lambda_handler:16","message":"Pydantic version -> 2.0","timestamp":"2023-07-04 10:31:21,050+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:18","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:31:21,051+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:18","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:31:21,051+0000","service":"service_undefined"}
END RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631
REPORT RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631	Init Duration: 0.08 ms	Duration: 362.40 ms	Billed Duration: 363 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

Thank you.

[leandrodamascena]

I forgot to mention that the refactored model code works in Pydantic v1 and v2. It's still too early to come to any conclusions and so I'm going to write the RFC. But that's what we intend to allow a seamless experience for customers who choose to use Pydantic v1 or v2.

❯ sam local invoke --event events/sqs.json
Invoking app.lambda_handler (python3.10)
Local image is up-to-date
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.

Mounting /home/leandro/DEVEL-PYTHON/tmp/pydantic2/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 7b6d3090-e61f-4a36-8478-d17a3b16ed4e Version: $LATEST
START RequestId: 7b6d3090-e61f-4a36-8478-d17a3b16ed4e Version: $LATEST
{"level":"INFO","location":"lambda_handler:13","message":"Pydantic version -> 1.10.10","timestamp":"2023-07-04 10:39:45,260+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:15","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:39:45,261+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:15","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:39:45,261+0000","service":"service_undefined"}
END RequestId: 7b6d3090-e61f-4a36-8478-d17a3b16ed4e
REPORT RequestId: 7b6d3090-e61f-4a36-8478-d17a3b16ed4e	Init Duration: 0.14 ms	Duration: 211.51 ms	Billed Duration: 212 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	

[half2me]

Thanks a lot! I'm happy to test this it once its available and provide feedback

[Wurstnase]

Please keep in mind, that some old feature still work but are deprecated.
e.g. root_validator

Pydantic V1 style @root_validator validators are deprecated. You should migrate to Pydantic V2 style @model_validator validators, see the migration guide for more details

[leandrodamascena]

Please keep in mind, that some old feature still work but are deprecated. e.g. root_validator

Thanks for bringing this up @Wurstnase. I am considering this note in the RFC.

[heitorlessa]

Initial RFC is out by @leandrodamascena: #2672

[ran-isenberg]

Thanks @leandrodamascena for this awesome summary!

I've noticed some extra issues in the pydantic migration guide that require our attention:

1. the parse_raw/ parse_obj funcs are now deprecated and need to renamed
2. Types: Empty Dicts/List now fail validation, they didnt before so this might alter people's validations
3. .dict() and .json() are also removed, we use in several use cases, one of them is also idempotency and even the parser docs.
4. @Validators However, in Pydantic V2, when a TypeError is raised in a validator, it is no longer converted into a ValidationError, so that means people need to change their code to catch TypeError (breaking change) OR we change the validator to raise ValidationError instead of TypeError.
5. Issue number 4 also applies to code such as json.loads that raises TypeError, it's best to catch all exceptions and perhaps re-raise them as validationerrors

[leandrodamascena]

@ran-isenberg do you mind moving this conversation to the RFC issue? So we can address all comments and update the RFC.

RFC: #2672

[heitorlessa]

Lê - could you add a table to the RFC at the bottom of what’s deprecated vs removed? And behaviour changes affecting us like validation error Ran brought up? It’d give us more confidence that we’re not missing anything, and don’t overengineer for something that isn’t removed in V2 (yet).

…

On Wed, 5 Jul 2023 at 19:39, Ran Isenberg ***@***.***> wrote: Thanks @leandrodamascena <https://github.com/leandrodamascena> for this awesome summary! I've noticed some extra issues in the pydantic migration guide that require our attention: 1. the parse_raw/ parse_obj funcs are now deprecated and need to renamed 2. Types: Empty Dicts/List now fail validation, they didnt before so this might alter people's validations 3. .dict() and .json() are also removed, we use in several use cases, one of them is also idempotency and even the parser docs. 4. @Validators <https://github.com/Validators> However, in Pydantic V2, when a TypeError is raised in a validator, it is no longer converted into a ValidationError, so that means people need to change their code to catch TypeError (breaking change) OR we change the validator to raise ValidationError instead of TypeError. 5. #4 <#4> also applied to code such as json.loads that raises TypeError, it's best to catch all exceptions and perhas re-raise them as validationerrors — Reply to this email directly, view it on GitHub <#2427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBF63U2YLU5XHERX5N3XOWRERANCNFSM6AAAAAAY7V5Q2M> . You are receiving this because you commented.Message ID: ***@***.*** com>

[leandrodamascena]

@heitorlessa yes, great idea Heitor! 🚀

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.