Feature request: Pydantic v2 managed layer

[lmmx]

Use case

I saw that there was support coming for Pydantic v2 in lambda powertools:

• feat(layers): add support for publishing v2 layer #1558
• feat(parser): add support for Pydantic v2 #2733

but it seems like it is only support, not incorporation into the library dependencies (i.e. the library is not bumping to v2). This conflicts with the desired workflow of using new versions of the layer in combination with up to date Pydantic models.

Solution/User Experience

Will you consider publishing managed layers with v2 of Pydantic as the dependency?

This seems in line with the linked 'Tenet' of adopting best practices:

Eases the adoption of best practices. The main priority of the utilities is to facilitate best practices adoption, as defined in the AWS Well-Architected Serverless Lens; all other functionality is optional.

It could also be done in a backward compatible way if it was published with a separate ARN

We strive for backwards compatibility. New features and changes should keep backwards compatibility. If a breaking change cannot be avoided, the deprecation and migration process should be clearly defined.

It also relates to 'following language idioms and community common practices':

Progressive. Utilities are designed to be incrementally adoptable for customers at any stage of their Serverless journey. They follow language idioms and their community’s common practices.

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hello @lmmx! Thank you for opening this issue. I am currently reading this to respond to your points properly.

[leandrodamascena]

Hi @lmmx , thank you so much for bringing this issue to discussion, this is an important point.

Here in Powertools we do our best to make clear customer communication in every release we make and new features we add. We follow semantic versioning and therefore breaking changes are scheduled for major releases and even then with early and clear communication to our customers. That said, we've kept our layer with Pydanticv1 so that our users can upgrade to Pydanticv2 as often as they like, without forcing it and causing supply chain problems, as we experienced during the transition.

We are looking to add Pydantic V2 as our default in Powertools v3 - tentatively end of the year (https://docs.powertools.aws.dev/lambda/python/latest/roadmap/). Creating a new layer at this moment with unique support for Pydantic v2 adds more complexity to our process for maintaining and updating that layer. At this time, we are working hard to implement new features such as external observability providers and sensitive data masking.

To keep our eyes on the needs of the community, I added the need-customer-feedback and revisit-in-3-months labels - who knows, this might just be what other customers are looking for too.

Thanks.

[lmmx]

Hey Leandro that's cool, I built a layer myself (it didn't take much effort) so this isn't urgent. I found a link to an AWS repo which linked to another one which had a Dockerfile, and I got a general idea of how to use rustup which is needed for Pydantic v2.

I suspect I actually end up installing Rust twice but it works so it's all good.

Python 3.10 layer Dockerfile is below, with apologies for any mess, in case it helps anyone else but I know you have your own superior CDK approach 😺

Click to show Dockerfile

FROM mlupin/docker-lambda:python3.10-build AS build

USER root

ENV SCRATCH_DIR=/var/task/scratch
ENV LAYER_ZIP="pydantic_layer.zip"

WORKDIR /var/task

# https://towardsdatascience.com/how-to-shrink-numpy-scipy-pandas-and-matplotlib-for-your-data-product-4ec8d7e86ee4
ENV CFLAGS "-g0 -Wl,--strip-all -DNDEBUG -Os -I/usr/include:/usr/local/include -L/usr/lib64:/usr/local/lib64:/usr/lib:/usr/local/lib"

# Rust up https://github.com/rust-serverless/lambda-rust/blob/master/Dockerfile
# via https://github.com/awslabs/aws-lambda-rust-runtime#25-docker
# via https://docs.aws.amazon.com/lambda/latest/dg/lambda-rust.html
ENV RUST_VERSION=1.70.0
RUN yum install -y jq openssl-devel gcc zip
RUN set -o pipefail && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
    | CARGO_HOME=/cargo RUSTUP_HOME=/rustup sh -s -- -y --profile minimal --default-toolchain $RUST_VERSION
# ENV PATH="/root/.cargo/bin:${PATH}"
ENV PATH="/cargo/bin:${PATH}"

RUN rustup default $RUST_VERSION

RUN ls /cargo/bin && \
    rustc --version

# ------------------- END OF SYSTEM BUILD DEPENDENCY SETUP ------------------

RUN mkdir -p $SCRATCH_DIR/python && \
    mkdir -p $SCRATCH_DIR/lib

# Install build dependencies for the wheels
RUN python3.10 -m pip install --upgrade pip && \
    python3.10 -m pip --version

# ------------------- END OF PACKAGE BUILD DEPENDENCY SETUP ------------------

ENV PYDANTIC_VERSION=2.1.1

# Download pydantic source distribution
RUN python3.10 -m pip download --no-binary=:all: pydantic==$PYDANTIC_VERSION

# Extract the pydantic package and build the wheel
RUN ls && tar xzf pydantic-$PYDANTIC_VERSION.tar.gz 

WORKDIR /var/task/pydantic-$PYDANTIC_VERSION

RUN python3.10 -m pip install build
RUN python3.10 -m build

WORKDIR /var/task

ENV BUILT_WHEEL=pydantic-$PYDANTIC_VERSION/dist/*-$PYDANTIC_VERSION-*.whl

RUN ls $BUILT_WHEEL

# ------------------------ END OF PYDANTIC BUILD --------------------------------

# Install the wheels with pip
# (Note: previously this used --compile but now we already did the wheel compilation)
RUN python3.10 -m pip install --no-compile --no-cache-dir \
    -t $SCRATCH_DIR/python \
    $BUILT_WHEEL && \
    ls $SCRATCH_DIR/python

# -------------------- END OF PACKAGE INSTALLATION -----------------------------

# Clean up the sdist and wheel
RUN rm pydantic-$PYDANTIC_VERSION.tar.gz && \
    rm -r pydantic-$PYDANTIC_VERSION

RUN cd $SCRATCH_DIR \
    && zip -r9 $LAYER_ZIP python  \
    && zip -r9 $LAYER_ZIP lib
    
FROM mlupin/docker-lambda:python3.10-build
 
COPY --from=build /var/task/scratch /opt
COPY --from=build /var/task /var/task

RUN PYTHONPATH=/opt/python python3 -c 'import pydantic'

Of course there are features in Powertools including Pydantic integration which go beyond Pydantic's base library, so I'm still looking forward to the managed layer.

[leandrodamascena]

Hello @lmmx! Wow, this Dockfile is priceless!! I'm going to copy and save it because I might need it soon to create some tests. Thank you so much for sharing this.

Yes, we are working to merge the existing PR and start working on the V3 plan as next steps.

I'll update this issue in case of news, ok?

Thanks