No module named 'pydantic_core._pydantic_core' using AWS Lambdas #6557

quirozs · 2023-07-10T16:20:08Z

Initial Checks

I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description

I have pydantic 2.0.2 installed into my AWS Lambda with python3.10 and whenever I try to import pydantic I get the following exception:

[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'pydantic_core._pydantic_core'
Traceback (most recent call last):

I already checked that the module exists and it is accessible from python path. It was working with pydantic v1.

Example Code

No response

Python, Pydantic & OS Version

I can't import pydantic but the directory shows:

pydantic-2.0.2.dist-info
pydantic_core-2.1.2.dist-info

Selected Assignee: @davidhewitt

The text was updated successfully, but these errors were encountered:

johnbley · 2023-07-10T16:37:14Z

I'm seeing this as well with fastapi-0.100.0, pydantic-2.0.2, and pydantic-core-2.1.2:

File "/var/task/fastapi/params.py", line 5, in <module>
from pydantic.fields import FieldInfo
File "/var/task/pydantic/__init__.py", line 3, in <module>
import pydantic_core
File "/var/task/pydantic_core/__init__.py", line 6, in <module>
from ._pydantic_core import (
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'

samuelcolvin · 2023-07-10T16:39:58Z

I asked about this on twitter and people said it was working fine for them - https://twitter.com/pydantic/status/1678298418983124992.

samuelcolvin · 2023-07-10T16:42:26Z

Weird that pydantic is being installed from the sdist, not the wheel. If you don't have the rust toolchain available to build pydantic-core, and you're not installing a wheel, it makes sense that you can't import pydantic_core._pydantic_core - that's the binary shared library.

Maybe it's related to which architecture you're using? Could you give more information about how the lambda function is being
called?

johnbley · 2023-07-10T16:44:44Z

It's also quite odd that fastapi (0.100.0)'s requirements are "pydantic>=1.7.4,!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0",....

samuelcolvin · 2023-07-10T16:47:59Z

not really, they're trying to support as many versions of pydantic as possible - FastAPI 0.100 has support for both Pydantic V1 and Pydantic V2 - @tiangolo did an amazing job with this, so you can upgrade fastapi without having to upgrade pydantic yet.

quirozs · 2023-07-10T16:52:06Z

Maybe it's related to which architecture you're using? Could you give more information about how the lambda function is being called?

Forgot to mention the Lambda is running in arm64, the pydantic library was installed using a docker running the official python3.10 arm64 image. I'm calling the lambda function directly from the AWS console using a test event

johnbley · 2023-07-10T16:59:05Z

I'm on x86_64/amd64. My lambda isn't using pydantic directly if that helps.

samuelcolvin · 2023-07-10T17:51:48Z

Can you show the log from building/installing python packages.

quirozs · 2023-07-10T19:35:27Z

Sure, I'm using CDK to build the lambda function

Collecting redis==4.5.5
  Downloading redis-4.5.5-py3-none-any.whl (240 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.3/240.3 kB 17.4 MB/s eta 0:00:00
Collecting pydantic==2.0.2
  Downloading pydantic-2.0.2-py3-none-any.whl (359 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 359.1/359.1 kB 249.1 MB/s eta 0:00:00
Collecting async-timeout>=4.0.2
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting pydantic-core==2.1.2
  Downloading pydantic_core-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 5.7 MB/s eta 0:00:00
Collecting annotated-types>=0.4.0
  Downloading annotated_types-0.5.0-py3-none-any.whl (11 kB)
Collecting typing-extensions>=4.6.1
  Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Installing collected packages: typing-extensions, async-timeout, annotated-types, redis, pydantic-core, pydantic
Successfully installed annotated-types-0.5.0 async-timeout-4.0.2 pydantic-2.0.2 pydantic-core-2.1.2 redis-4.5.5 typing-extensions-4.7.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip

samuelcolvin · 2023-07-10T19:50:47Z

Well there's you're issue - you're installing the x86_64 wheel but trying to run it on amd64.

I've no what CDK is, but it/they most have instructions on how to "cross install" binary packages.

Try asking on their GitHub. You might also try Googling how they cope with other binary packages like cryptography, pillow, bcrypt etc.

quirozs · 2023-07-10T19:54:52Z

Oh wow, that's right. Thanks for your help

davidhewitt · 2023-07-10T19:57:11Z

There may be overlapping issues here, @johnbley is reporting "x86_64/amd64" (which afaik are the same thing) and @quirozs is reporting arm64.

johnbley · 2023-07-10T23:38:12Z

Yes, I'm definitely not using cross-built binaries, and it seems to be getting an appropriate binary for the AWS lambda python3.8 runtime:

Collecting fastapi>=0.42.0
   Downloading fastapi-0.100.0-py3-none-any.whl (65 kB)
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4
   Downloading pydantic-2.0.2-py3-none-any.whl (359 kB)
Collecting pydantic-core==2.1.2
   Downloading pydantic_core-2.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)

oriori1703 · 2023-07-11T07:19:45Z

@johnbley it seems the pydantic-core wheel is specifically for python 3.7 (cp37), while you said you are running python 3.8

davidhewitt · 2023-07-11T08:28:34Z

We do ship wheels for all python versions, possibly a mismatch between environment during build and runtime?

johnbley · 2023-07-11T13:19:16Z

@johnbley it seems the pydantic-core wheel is specifically for python 3.7 (cp37), while you said you are running python 3.8

Yep, that is it. I'll update the build environment to match versions. Thanks for your help!

Aarif1430 · 2023-07-13T07:32:27Z

I encountered a similar problem this week, and I found that changing the runtime from Python 3.9 to Python 3.10 resolved it. I have attached a screenshot of the AWS runtime settings below for your reference.

Esquive · 2023-07-18T20:07:24Z

@quirozs I have the same issue. I use the official AWS lambda arm64 python 3.10 image to assemble the dependencies with platform set to arm64.

Collecting pydantic-core==2.3.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->fastapi>=0.100.0->-r /tmp/requirements.txt (line 2)) Obtaining dependency information for pydantic-core==2.3.0 from https://files.pythonhosted.org/packages/2d/1e/d7ef26e565273dd26c0309244f43c0ec555ed4ba9ef83d794967b432c4c9/pydantic_core-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata Downloading pydantic_core-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.1 kB)

What did you do change from arm64 runtime to x86_64? I would like to stay on ARM though :-)

davidhewitt · 2023-07-18T20:37:53Z

@Esquive how are you building your lambda? You should be able to install the arm wheel instead of the x86_64 one.

Esquive · 2023-07-18T21:05:55Z

@davidhewitt I create my docker image:

FROM --platform=linux/arm64 amazon/aws-lambda-python:3.10

ARG github_token
ENV GITHUB_TOKEN=$github_token

COPY requirements.txt /tmp

RUN python -m pip install --upgrade pip

And run the dependencies install:

docker build --platform=linux/arm64 --build-arg var_name=${VARIABLE_NAME} -t armlinuxpy310 .
docker run -itd --name deps_builder -v $(shell pwd)/layers:/tmp/deps -e GITHUB_TOKEN armlinuxpy310 /bin/sh
docker exec deps_builder python -m pip install -t /tmp/deps/python -r /tmp/requirements.txt

The output from the container however displays the x86_64 versions of pydantic core has been pulled.

davidhewitt · 2023-07-19T08:42:31Z

According to the docker image registry your base image is an x86_64 (amd64) image:

I'm not entirely sure how your image is working (emulation?) though I suspect that the python inside is identifying itself as x86_64, hence the wrong wheels getting pulled. It looks like FROM --platform=linux/arm64 doesn't error if the image isn't multiplatform, which is unfortunate.

Can you retry with public.ecr.aws/lambda/python:3.10 as your base image instead? That is a proper multiplatform manifest:

david@david-pc:~$ docker buildx imagetools inspect amazon/aws-lambda-python:3.10
Name:      docker.io/amazon/aws-lambda-python:3.10
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:3914f5a0e3788a851feabbe9ba2c304ec9fd981fcd2d955644fa853634baa69d

----

david@david-pc:~$ docker buildx imagetools inspect public.ecr.aws/lambda/python:3.10
Name:      public.ecr.aws/lambda/python:3.10
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:c12570172ee77c2f5f21e776cfa8f42b01d428ea64e2fb80b4bc47aecff34fd6

Manifests:
  Name:      public.ecr.aws/lambda/python:3.10@sha256:b776f3d5894f7fe5daa82fea7850b085299d35d2a17a705c5067c64bdf9565d4
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

  Name:      public.ecr.aws/lambda/python:3.10@sha256:3914f5a0e3788a851feabbe9ba2c304ec9fd981fcd2d955644fa853634baa69d
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

Esquive · 2023-07-19T13:12:57Z

Hello @davidhewitt thanks for your answer, I just came here to tell you I figured it out to avoid you spending time on this. You are correct with the docker image. To really make sure:

FROM --platform=linux/arm64 public.ecr.aws/lambda/python:3.10-arm64

On top of that I realised that docker was raising a warning on the architecture mismatch when not adding the platform option:

docker run --platform=linux/arm64 -itd --name deps_builder -v $(shell pwd)/layers:/tmp/deps -e GITHUB_TOKEN armlinuxpy310 /bin/sh

With the both above I was sure to be on an emulated arm64 environment and I pulled the correct pydantic packages.

25thbamofthetower · 2023-07-20T01:08:51Z

@davidhewitt I got this error with fastapi==0.100.0 and Python 3.8 on AWS lambda. I confirm it's using x86_64 architecture, and if I switch back to fastapi==0.99.1 then I don't get the error. Any suggestions on how to fix this?

davidhewitt · 2023-07-20T07:53:18Z

@25thbamofthetower can you share detail about how you are building your lambda?

paalvibe · 2023-07-21T09:02:06Z

@25thbamofthetower can you share detail about how you are building your lambda?

We have two generations of amplify apps.

One app is pushed manually with amplify push on the developer laptop. amplify push will build a lambda and deploy it to AWS. It uses several config files. One of them is cloudformation-template.json file.

The newer generation will build and deploy the app, including the lambda, automatically on commits to a specific git branch, e.g. main.

Here is more information: https://docs.amplify.aws/cli/function/#function-templates

This is the relevant section of it of the cloudformation template json:

  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {
        "aws:asset:path": "./src",
        "aws:asset:property": "Code"
      },
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "deploymentBucketName"
          },
          "S3Key": {
            "Ref": "s3Key"
          }
        },
        "Handler": "index.handler",
        "FunctionName": {
          "Fn::If": [
            "ShouldNotCreateEnvResources",
            "fooapi",
            {
              "Fn::Join": [
                "",
                [
                  "fooapi",
                  "-",
                  {
                    "Ref": "env"
                  }
                ]
              ]
            }
          ]
        },
        "Environment": {
          "Variables": {
            "ENV": {
              "Ref": "env"
            },
            "REGION": {
              "Ref": "AWS::Region"
            }
          }
        },
        "Role": {
          "Fn::GetAtt": [
            "LambdaExecutionRole",
            "Arn"
          ]
        },
        "Runtime": "python3.9",
        "Layers": [],
        "Timeout": 25
      }
    },

25thbamofthetower · 2023-07-22T17:12:03Z

@davidhewitt davidhewitt

I use AWS CDK to build and deploy the lambda and api gateway

I don't think it's this, because switching back to fastapi==0.99.1 works without any changes to the CDK infrastructure code.

As for the Python code, I'm using bare minimum fastapi (https://github.com/tiangolo/fastapi) and mangum (https://github.com/jordaneremieff/mangum).

lober-io · 2023-12-20T11:17:52Z

I'm facing the same issue trying to write a lambda function within AWS Amplify.

I coded my lambda using python 3.10, it uses a package that has inner dependencies with:
pydantic==2.5.2
pydantic-core==2.14.5
as per AWS Amplify doc I used pipenv to create Pipfile and Pipfile.lock. I built and deployed the function running amplify push but when I tried to test it I got:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'pydantic_core._pydantic_core' Traceback (most recent call last):

I got the same error running amplify push on macOS and Windows, even if, as expected, Pipfile and Pipfile.lock did not changed.

To the best of my knowledge I cannot find an equivalent of pip --platform as mentioned by @davidhewitt here to be used with pipenv.

Any suggestion is more than welcomed!

I had a similar issue and figured out that they old package was still included in the Lambda build...
Deleting everything below /Users/{USERNAME}/.local/share/virtualenvs/ solved my issue

mohit2152sharma · 2023-12-23T08:59:10Z

For anyone facing this issues with serverless and aws lambda. Following may help. This issue is due to different architectures.
I was using public.ecr.aws/sam/build-python3.11:latest, this image comes with arm64 architecture. You can confirm this with docker image inspect public.ecr.aws/sam/build-python3.11:latest | grep "Architecture".
You can use the x86_64 image by using public.ecr.aws/sam/build-python3.11:latest-x86_64

daniellersisk · 2024-01-03T03:23:26Z

Just solved this error. For me the issue was occurring because I was using python 3.12 in AWS and in my layer. Once I switched both of them to 3.10 all worked just fine.

ResearcherSara · 2024-01-11T04:53:41Z

Just noticed that Amazon has documentation on this which recommends passing the --platform flag to pip: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries
(This is the alternative to using pip in docker which I suggested above.)

This solution work out for me, thank you for this reference 😊.

fastapi==0.104.1 pydantic==2.5.2

Can you share what did you do? I checked the link, but couldnt find out. in importing openai, I faced with this error. really dont have any idea about it.

pierluigi-failla · 2024-01-11T10:28:47Z

I'm facing the same issue trying to write a lambda function within AWS Amplify.

I coded my lambda using python 3.10, it uses a package that has inner dependencies with:
pydantic==2.5.2
pydantic-core==2.14.5
as per AWS Amplify doc I used pipenv to create Pipfile and Pipfile.lock. I built and deployed the function running amplify push but when I tried to test it I got:

[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'pydantic_core._pydantic_core' Traceback (most recent call last):

I got the same error running amplify push on macOS and Windows, even if, as expected, Pipfile and Pipfile.lock did not changed.

To the best of my knowledge I cannot find an equivalent of pip --platform as mentioned by @davidhewitt here to be used with pipenv.

Any suggestion is more than welcomed!

I solved this using a Lambda Layer with all my deps, also in order to be sure about platforms and architecture I created it using a Cloud9 instance.

lober-io · 2024-01-11T10:51:03Z

For amplify you have no other option than this or running the build only in the pipeline with the right architecture (instead on your local environment)

I'm facing the same issue trying to write a lambda function within AWS Amplify.
I coded my lambda using python 3.10, it uses a package that has inner dependencies with:
pydantic==2.5.2
pydantic-core==2.14.5
as per AWS Amplify doc I used pipenv to create Pipfile and Pipfile.lock. I built and deployed the function running amplify push but when I tried to test it I got:
[ERROR] Runtime.ImportModuleError: Unable to import module 'index': No module named 'pydantic_core._pydantic_core' Traceback (most recent call last):
I got the same error running amplify push on macOS and Windows, even if, as expected, Pipfile and Pipfile.lock did not changed.
To the best of my knowledge I cannot find an equivalent of pip --platform as mentioned by @davidhewitt here to be used with pipenv.
Any suggestion is more than welcomed!
I solved this using a Lambda Layer with all my deps, also in order to be sure about platforms and architecture I created it using a Cloud9 instance.

Vanshikashah318 · 2024-01-18T04:44:15Z

I am writing a lambda function with openai as well and I have used a lambda layer where I pip installed openai on my local machine zipped the file and uploaded it as a layer it has pydantic core I still keep facing the same issue. I have chnaged the versions from 3.12 to 3.10 to even 3.8 but nothing seems to work at all. What do I do?

mohit2152sharma · 2024-01-18T06:29:46Z

@Vanshikashah318 , I would recommend to check your system's and lambda's architecture

For anyone facing this issues with serverless and aws lambda. Following may help. This issue is due to different architectures. I was using public.ecr.aws/sam/build-python3.11:latest, this image comes with arm64 architecture. You can confirm this with docker image inspect public.ecr.aws/sam/build-python3.11:latest | grep "Architecture". You can use the x86_64 image by using public.ecr.aws/sam/build-python3.11:latest-x86_64

Vanshikashah318 · 2024-01-18T20:11:53Z

@Vanshikashah318 , I would recommend to check your system's and lambda's architecture

For anyone facing this issues with serverless and aws lambda. Following may help. This issue is due to different architectures. I was using public.ecr.aws/sam/build-python3.11:latest, this image comes with arm64 architecture. You can confirm this with docker image inspect public.ecr.aws/sam/build-python3.11:latest | grep "Architecture". You can use the x86_64 image by using public.ecr.aws/sam/build-python3.11:latest-x86_64

I tried changing the architecture to arm64 also for both the lambda func as well as the layer but still the same issue. Any other suggestions what I could do I am also working on 3.12.

NateLeeP · 2024-01-24T14:27:10Z

Just noticed that Amazon has documentation on this which recommends passing the --platform flag to pip: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries
(This is the alternative to using pip in docker which I suggested above.)

This solution work out for me, thank you for this reference 😊.
fastapi==0.104.1 pydantic==2.5.2

Can you share what did you do? I checked the link, but couldnt find out. in importing openai, I faced with this error. really dont have any idea about it.

In order to install pydantic-core I executed the AWS Documentation recommended command
pip install
--platform manylinux2014_x86_64
--target=package
--implementation cp
--python-version 3.x
--only-binary=:all: --upgrade
<package_name>

However, I had to add an additional platform name: 'manylinux_2_17_x86_64'. 'pydantic-core' appears to have a non-standard / misleading naming structure for uploading files. While the files are suffixed with 'manylinux2014_x86_64', the pip installer did not register it as the 'platform' name. It needed 'manylinux_2_17_x86_64' as the platform name. This installed it in a version that was compatiable with my x86_64, python version 3.9 Lambda.

intoxicated · 2024-02-06T08:56:20Z

Hi Recently we have had the same problem deploying with serverless, error faces in the same way, error messages related with pydantic-core.

The solution was set python runtime version in layer declaration.

service: app
plugins:
- serverless-python-requirements
- serverless-localstack
provider:
  name: aws
  environment:
    STAGE: ${self:provider.stage, self:custom.defaultStage}
    runtime: "python3.11"
  memorySize: 128
  timeout: 30
  stage: ${opt:stage, "dev"}
  region: us-west-1
  deploymentBucket:
    name: the-bucket-xx

package:
  individually: true
  patterns: {}
custom:
  pythonRequirements:
    usePoetry: true
    dockerizePip: true
    slim: true
    layer:
      name: the-layer-xxx
      description: Dependencies of app
      compatibleRuntimes:
        - "python3.11"
    useDownloadCache: true
    useStaticCache: true
  defaultStage: "local"

Packages versions:

pydantic = 2.3.0
pydantic-settings = 2.0.3
pydantic-core = 2.6.0
fastapi = 0.100.0

That fix the pydantic-core issue

Your setting doesn't have anything regards architecture :| this issue is related to architecture mismatch problem

intoxicated · 2024-02-06T09:34:01Z

-MacBook-Pro fastapi-serverless % pip install \           
--platform manylinux2014_aarch64 \
--target=package \
--implementation cp \
--python-version 3.11 \
--only-binary=:all: --upgrade \
"fastapi[all]"
Collecting fastapi[all]
  Obtaining dependency information for fastapi[all] from https://files.pythonhosted.org/packages/bf/97/60351307ab4502908d29f64f2801a36709a3f1888447bb328bc373d6ca0e/fastapi-0.109.2-py3-none-any.whl.metadata
  Using cached fastapi-0.109.2-py3-none-any.whl.metadata (25 kB)
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi[all])
  Obtaining dependency information for pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 from https://files.pythonhosted.org/packages/db/dc/afecbd9650f486889181c6d1a0d675b580c06253ea7e304588e4c7485bdb/pydantic-2.6.1-py3-none-any.whl.metadata
  Using cached pydantic-2.6.1-py3-none-any.whl.metadata (83 kB)
Collecting starlette<0.37.0,>=0.36.3 (from fastapi[all])
  Obtaining dependency information for starlette<0.37.0,>=0.36.3 from https://files.pythonhosted.org/packages/eb/f7/372e3953b6e6fbfe0b70a1bb52612eae16e943f4288516480860fcd4ac41/starlette-0.36.3-py3-none-any.whl.metadata
  Using cached starlette-0.36.3-py3-none-any.whl.metadata (5.9 kB)
Collecting typing-extensions>=4.8.0 (from fastapi[all])
  Obtaining dependency information for typing-extensions>=4.8.0 from https://files.pythonhosted.org/packages/b7/f4/6a90020cd2d93349b442bfcb657d0dc91eee65491600b2cb1d388bc98e6b/typing_extensions-4.9.0-py3-none-any.whl.metadata
  Using cached typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB)
Collecting email-validator>=2.0.0 (from fastapi[all])
  Obtaining dependency information for email-validator>=2.0.0 from https://files.pythonhosted.org/packages/09/68/d237a603d524ba052e292d71c89939bfa70e3ec7963b255cc3ef7a8770a0/email_validator-2.1.0.post1-py3-none-any.whl.metadata
  Using cached email_validator-2.1.0.post1-py3-none-any.whl.metadata (25 kB)
Collecting httpx>=0.23.0 (from fastapi[all])
  Obtaining dependency information for httpx>=0.23.0 from https://files.pythonhosted.org/packages/39/9b/4937d841aee9c2c8102d9a4eeb800c7dad25386caabb4a1bf5010df81a57/httpx-0.26.0-py3-none-any.whl.metadata
  Using cached httpx-0.26.0-py3-none-any.whl.metadata (7.6 kB)
Collecting itsdangerous>=1.1.0 (from fastapi[all])
  Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting jinja2>=2.11.2 (from fastapi[all])
  Obtaining dependency information for jinja2>=2.11.2 from https://files.pythonhosted.org/packages/30/6d/6de6be2d02603ab56e72997708809e8a5b0fbfee080735109b40a3564843/Jinja2-3.1.3-py3-none-any.whl.metadata
  Using cached Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB)
Collecting orjson>=3.2.1 (from fastapi[all])
  Obtaining dependency information for orjson>=3.2.1 from https://files.pythonhosted.org/packages/af/de/81428e091916e46d1b0b5501c07f52ae7b1f6d38242e9c3ce8778c00f529/orjson-3.9.13-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading orjson-3.9.13-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.4/49.4 kB 129.8 kB/s eta 0:00:00
Collecting pydantic-extra-types>=2.0.0 (from fastapi[all])
  Obtaining dependency information for pydantic-extra-types>=2.0.0 from https://files.pythonhosted.org/packages/d2/1e/8c9021519140f813a6b4d78cd4257b3290c559859bf96a858936acc2e78a/pydantic_extra_types-2.5.0-py3-none-any.whl.metadata
  Using cached pydantic_extra_types-2.5.0-py3-none-any.whl.metadata (2.6 kB)
Collecting pydantic-settings>=2.0.0 (from fastapi[all])
  Obtaining dependency information for pydantic-settings>=2.0.0 from https://files.pythonhosted.org/packages/5d/c9/8042368e9a1e6e229b5ec5d88449441a3ee8f8afe09988faeb190af30248/pydantic_settings-2.1.0-py3-none-any.whl.metadata
  Using cached pydantic_settings-2.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting python-multipart>=0.0.7 (from fastapi[all])
  Obtaining dependency information for python-multipart>=0.0.7 from https://files.pythonhosted.org/packages/94/35/142fff3d85da49377ada6936ad9b776263549ab22656969b2fcd0bdb10f7/python_multipart-0.0.7-py3-none-any.whl.metadata
  Using cached python_multipart-0.0.7-py3-none-any.whl.metadata (2.5 kB)
Collecting pyyaml>=5.3.1 (from fastapi[all])
  Obtaining dependency information for pyyaml>=5.3.1 from https://files.pythonhosted.org/packages/5e/94/7d5ee059dfb92ca9e62f4057dcdec9ac08a9e42679644854dc01177f8145/PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.1 kB)
Collecting ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 (from fastapi[all])
  Obtaining dependency information for ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 from https://files.pythonhosted.org/packages/19/f3/1431713de9e5992e5e33ba459b4de28f83904233958855d27da820a101f9/ujson-5.9.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading ujson-5.9.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (8.7 kB)
Collecting uvicorn[standard]>=0.12.0 (from fastapi[all])
  Obtaining dependency information for uvicorn[standard]>=0.12.0 from https://files.pythonhosted.org/packages/c7/f3/29caa83f5795b20ed3aca357c648f3ae995ff6ff08e38b22387017abbdc5/uvicorn-0.27.0.post1-py3-none-any.whl.metadata
  Using cached uvicorn-0.27.0.post1-py3-none-any.whl.metadata (6.4 kB)
Collecting dnspython>=2.0.0 (from email-validator>=2.0.0->fastapi[all])
  Obtaining dependency information for dnspython>=2.0.0 from https://files.pythonhosted.org/packages/b6/83/4a684a63d395007670bc95c1947c07045fe66141574e2f7e9e347df8499a/dnspython-2.5.0-py3-none-any.whl.metadata
  Using cached dnspython-2.5.0-py3-none-any.whl.metadata (5.8 kB)
Collecting idna>=2.0.0 (from email-validator>=2.0.0->fastapi[all])
  Obtaining dependency information for idna>=2.0.0 from https://files.pythonhosted.org/packages/c2/e7/a82b05cf63a603df6e68d59ae6a68bf5064484a0718ea5033660af4b54a9/idna-3.6-py3-none-any.whl.metadata
  Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting anyio (from httpx>=0.23.0->fastapi[all])
  Obtaining dependency information for anyio from https://files.pythonhosted.org/packages/bf/cd/d6d9bb1dadf73e7af02d18225cbd2c93f8552e13130484f1c8dcfece292b/anyio-4.2.0-py3-none-any.whl.metadata
  Using cached anyio-4.2.0-py3-none-any.whl.metadata (4.6 kB)
Collecting certifi (from httpx>=0.23.0->fastapi[all])
  Obtaining dependency information for certifi from https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl.metadata
  Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx>=0.23.0->fastapi[all])
  Obtaining dependency information for httpcore==1.* from https://files.pythonhosted.org/packages/56/ba/78b0a99c4da0ff8b0f59defa2f13ca4668189b134bd9840b6202a93d9a0f/httpcore-1.0.2-py3-none-any.whl.metadata
  Using cached httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)
Collecting sniffio (from httpx>=0.23.0->fastapi[all])
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.23.0->fastapi[all])
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting MarkupSafe>=2.0 (from jinja2>=2.11.2->fastapi[all])
  Obtaining dependency information for MarkupSafe>=2.0 from https://files.pythonhosted.org/packages/1c/cf/35fe557e53709e93feb65575c93927942087e9b97213eabc3fe9d5b25a55/MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.0 kB)
Collecting annotated-types>=0.4.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi[all])
  Obtaining dependency information for annotated-types>=0.4.0 from https://files.pythonhosted.org/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl.metadata
  Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.16.2 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi[all])
  Obtaining dependency information for pydantic-core==2.16.2 from https://files.pythonhosted.org/packages/ba/cd/7344e100c17c4751224fabd6025ded7729e900ac5866febadec05b209479/pydantic_core-2.16.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading pydantic_core-2.16.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (6.5 kB)
Collecting python-dotenv>=0.21.0 (from pydantic-settings>=2.0.0->fastapi[all])
  Obtaining dependency information for python-dotenv>=0.21.0 from https://files.pythonhosted.org/packages/6a/3e/b68c118422ec867fa7ab88444e1274aa40681c606d59ac27de5a5588f082/python_dotenv-1.0.1-py3-none-any.whl.metadata
  Using cached python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Collecting click>=7.0 (from uvicorn[standard]>=0.12.0->fastapi[all])
  Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.12.0->fastapi[all])
  Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/76/7a/45c5a9a2e9d21f7381866eb7b6ead5a84d8fe7e54e35208eeb18320a29b4/httptools-0.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading httptools-0.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.6 kB)
Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.12.0->fastapi[all])
  Obtaining dependency information for uvloop!=0.15.0,!=0.15.1,>=0.14.0 from https://files.pythonhosted.org/packages/1f/c7/e494c367b0c6e6453f9bed5a78548f5b2ff49add36302cd915a91d347d88/uvloop-0.19.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading uvloop-0.19.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.9 kB)
Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.12.0->fastapi[all])
  Obtaining dependency information for watchfiles>=0.13 from https://files.pythonhosted.org/packages/0e/cf/126f0a8683f326d190c3539a769e45e747a80a5fcbf797de82e738c946ae/watchfiles-0.21.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading watchfiles-0.21.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.9 kB)
Collecting websockets>=10.4 (from uvicorn[standard]>=0.12.0->fastapi[all])
  Obtaining dependency information for websockets>=10.4 from https://files.pythonhosted.org/packages/6e/a4/51a25e591d645df71ee0dc3a2c880b28e5514c00ce752f98a40a87abcd1e/websockets-12.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Downloading websockets-12.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (6.6 kB)
Using cached email_validator-2.1.0.post1-py3-none-any.whl (32 kB)
Using cached httpx-0.26.0-py3-none-any.whl (75 kB)
Using cached httpcore-1.0.2-py3-none-any.whl (76 kB)
Using cached Jinja2-3.1.3-py3-none-any.whl (133 kB)
Downloading orjson-3.9.13-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (142 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.8/142.8 kB 655.9 kB/s eta 0:00:00
Using cached pydantic-2.6.1-py3-none-any.whl (394 kB)
Downloading pydantic_core-2.16.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 3.4 MB/s eta 0:00:00
Using cached pydantic_extra_types-2.5.0-py3-none-any.whl (21 kB)
Using cached pydantic_settings-2.1.0-py3-none-any.whl (11 kB)
Using cached python_multipart-0.0.7-py3-none-any.whl (22 kB)
Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (732 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 732.2/732.2 kB 24.9 MB/s eta 0:00:00
Using cached starlette-0.36.3-py3-none-any.whl (71 kB)
Using cached typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Downloading ujson-5.9.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (51 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.7/51.7 kB 3.8 MB/s eta 0:00:00
Using cached fastapi-0.109.2-py3-none-any.whl (92 kB)
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Using cached anyio-4.2.0-py3-none-any.whl (85 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached dnspython-2.5.0-py3-none-any.whl (305 kB)
Downloading httptools-0.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (324 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 324.8/324.8 kB 11.0 MB/s eta 0:00:00
Using cached idna-3.6-py3-none-any.whl (61 kB)
Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (29 kB)
Using cached python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading uvloop-0.19.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 9.5 MB/s eta 0:00:00
Downloading watchfiles-0.21.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 31.8 MB/s eta 0:00:00
Downloading websockets-12.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (131 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.5/131.5 kB 9.2 MB/s eta 0:00:00
Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Using cached uvicorn-0.27.0.post1-py3-none-any.whl (60 kB)
Installing collected packages: websockets, uvloop, ujson, typing-extensions, sniffio, pyyaml, python-multipart, python-dotenv, orjson, MarkupSafe, itsdangerous, idna, httptools, h11, dnspython, click, certifi, annotated-types, uvicorn, pydantic-core, jinja2, httpcore, email-validator, anyio, watchfiles, starlette, pydantic, httpx, pydantic-settings, pydantic-extra-types, fastapi
Successfully installed MarkupSafe-2.1.5 annotated-types-0.6.0 anyio-4.2.0 certifi-2024.2.2 click-8.1.7 dnspython-2.5.0 email-validator-2.1.0.post1 fastapi-0.109.2 h11-0.14.0 httpcore-1.0.2 httptools-0.6.1 httpx-0.26.0 idna-3.6 itsdangerous-2.1.2 jinja2-3.1.3 orjson-3.9.13 pydantic-2.6.1 pydantic-core-2.16.2 pydantic-extra-types-2.5.0 pydantic-settings-2.1.0 python-dotenv-1.0.1 python-multipart-0.0.7 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.36.3 typing-extensions-4.9.0 ujson-5.9.0 uvicorn-0.27.0.post1 uvloop-0.19.0 watchfiles-0.21.0 websockets-12.0

I believe I use arm64 version but still got error on my lambda :(

intoxicated · 2024-02-06T10:45:56Z

I guess it was something to do with cache (even though I cleaned it up)

For whom had similar issue, I was able to build and deploy successfully from scratch. And I had to add layer manually (i dont know why) in aws console.

Here is my serverless.yml

service: fastapi-serverless
frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.9
  architecture: arm64

functions:
  api:
    handler: app/main.handler
    events:
      - httpApi: '*'

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux
    layer:
      name: fastapi-dep
      description: fast dependencies
      compatibleRuntimes:
        - python3.9
      compatibleArchitectures:
        - arm64

Inthuson · 2024-02-06T10:59:33Z

Hi All, I am using the following CDK code and facing the same issue, except with Python 3.12

return _lambda.LayerVersion(
            self,
            "CommonPythonLambdaLayer",
            code=_lambda.Code.from_asset(
                build_dir,
                bundling=BundlingOptions(
                    image=_lambda.Runtime.PYTHON_3_12.bundling_image,
                    command=[
                        "bash",
                        "-c",
                        "pip install --no-cache -r requirements.txt -t /asset-output/python && cp -au . /asset-output/python",
                    ],
                    platform="linux/arm64",
                ),
            ),
            compatible_architectures=[_lambda.Architecture.ARM_64],
            compatible_runtimes=[_lambda.Runtime.PYTHON_3_12],
        )

The layer bundles some custom code and pydantic but I am receiving the following error whilst importing pydantic using the layer:
"errorMessage": "Unable to import module 'lambda_function': No module named 'pydantic_core._pydantic_core'"
Whilst inspecting the contents, although my function uses ARM, and I specified it everywhere the pydantic module installed is x86_64: _pydantic_core.cpython-312-x86_64-linux-gnu.so

I am using CDK pipelines, and I can see CodeBuild is using Image aws/codebuild/standard:7.0 , which is essentially Ubuntu, is there something obvious I am missing out?

sarflux · 2024-02-09T05:01:27Z

for anyone using windows I was able to solve this with the following command in powershell
pip install pydantic==2.6.1 --only-binary=:all: --upgrade --platform manylinux2014_x86_64 --implementation cp --python-version <python_runtime_version>

peterrauscher · 2024-02-09T23:52:55Z

for anyone using windows I was able to solve this with the following command in powershell pip install pydantic==2.6.1 --only-binary=:all: --upgrade --platform manylinux2014_x86_64 --implementation cp --python-version <python_runtime_version>

You rock, this worked for me. Python 3.12 runtime on Lambda, architecture x86_64. I only had to reinstall this one dependency too, everything else worked fine with "defaults".

bruceweir · 2024-02-11T00:07:52Z

@peterrauscher Was that as part of an openai API deployment?

peterrauscher · 2024-02-11T00:29:31Z

@peterrauscher Was that as part of an openai API deployment?

Nope, FastAPI + motor + Beanie

Good luck resolving your issue!

bruceweir · 2024-02-11T13:15:22Z

Finally got it working with Amplify.
Amplify (or pipenv) had apparently cached a non-native version of pydantic when I first set it up. Deleting the old lambda, and starting a fresh one worked once I had added a handcrafted lambda layer using...

 pip install --platform manylinux2014_x86_64 --target=python --implementation cp --python-version 3.10 --only-binary=:all: --upgrade openai

...on a linux box, zipping the content as per https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries and uploading it to AWS via the lambda layer console.
The lambda in Amplify was then updated to use that lambda layer (via its ARN).

andybbruno · 2024-02-16T14:25:07Z

Use this configuration if you're using MacOS and Serverless Framework

Reference: AWS Lambda Docs

custom:
  pythonRequirements:
      useStaticCache: false
      useDownloadCache: false
      pipCmdExtraArgs:
        - "--platform manylinux2014_x86_64"
        - "--implementation cp" 
        - "--python-version 3.11"
        - "--only-binary=:all:"
        - "--upgrade"

The big issue is that if you don't set useStaticCache: false and useDownloadCache: false, Serverless goes ahead and grabs MacOS cache files by default. So, when you deploy, you end up with binaries tailored for darwin instead of x86_64. Look at the pic below, you'll see the .so file having the .darwin tag.

ianbbqzy · 2024-02-27T06:27:50Z

Facing the issue even though I'm using arm locally on my mac and configure as such on Lambda

davidhewitt · 2024-02-27T11:00:40Z

To debug this issue: try checking the contents of the pydantic-core install on lambda.

Run the following code checks on startup of your lambda, before the import fails:

Check the contents of the installed pydantic-core package. Are the compiled library and its type stubs both present?
```
>>> from importlib.metadata import files
>>> [file for file in files('pydantic-core') if file.name.startswith('_pydantic_core')]
[PackagePath('pydantic_core/_pydantic_core.pyi'), PackagePath('pydantic_core/_pydantic_core.cpython-312-x86_64-linux-gnu.so')]
```
You should expect to see two files like those printed above. The compile library file will be a .so or .pyd with a name that varies according to the OS and Python version.
Check that your lambda's Python version is compatible with the compiled library version found above.
```
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-312-x86_64-linux-gnu.so'
```
You should expect to see the same suffix here as the compiled library, for example here we see this suffix .cpython-312-x86_64-linux-gnu.so indeed matches _pydantic_core.cpython-312-x86_64-linux-gnu.so.

If these two checks do not match, your build steps have not installed the correct native code for your lambda's target platform. You should adjust your build steps to change the version of the installed library which gets installed.

Most likely errors:

Your OS or CPU architecture is mismatched (e.g. darwin vs x86_64-linux-gnu). Try passing correct --platform argument to pip install when installing your lambda dependencies, or build inside a linux docker container for the correct platform.

Possible platforms at the moment are probably --platform manylinux2014_x86_64 or --platform manylinux2014_aarch64, but these may change with a future Pydantic major release.
Your Python version is mismatched (e.g. cpython-310 vs cpython-312). Try passing correct --python-version argument to pip install, or otherwise change the Python version used on your build.

MohammedAdain · 2024-03-09T06:30:38Z

To debug this issue: try checking the contents of the pydantic-core install on lambda.

Run the following code checks on startup of your lambda, before the import fails:
Check the contents of the installed pydantic-core package. Are the compiled library and its type stubs both present?
>>> from importlib.metadata import files
>>> [file for file in files('pydantic-core') if file.name.startswith('_pydantic_core')]
[PackagePath('pydantic_core/_pydantic_core.pyi'), PackagePath('pydantic_core/_pydantic_core.cpython-312-x86_64-linux-gnu.so')]
You should expect to see two files like those printed above. The compile library file will be a .so or .pyd with a name that varies according to the OS and Python version.
Check that your lambda's Python version is compatible with the compiled library version found above.
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-312-x86_64-linux-gnu.so'
You should expect to see the same suffix here as the compiled library, for example here we see this suffix .cpython-312-x86_64-linux-gnu.so indeed matches _pydantic_core.cpython-312-x86_64-linux-gnu.so.
If these two checks do not match, your build steps have not installed the correct native code for your lambda's target platform. You should adjust your build steps to change the version of the installed library which gets installed.

Most likely errors:

Your OS or CPU architecture is mismatched (e.g. darwin vs x86_64-linux-gnu). Try passing correct --platform argument to pip install when installing your lambda dependencies, or build inside a linux docker container for the correct platform.
Possible platforms at the moment are probably --platform manylinux2014_x86_64 or --platform manylinux2014_aarch64, but these may change with a future Pydantic major release.

Your Python version is mismatched (e.g. cpython-310 vs cpython-312). Try passing correct --python-version argument to pip install, or otherwise change the Python version used on your build.

Thanks @davidhewitt. This helped us troubleshoot our issue!

itsvaibhav01 · 2024-03-12T13:24:35Z

I had the same error in the AWS Lambda.
Solution: It worked when I changed the runtime to python3.10 on the lambda function.
❤️

shashidhar100 · 2024-03-15T19:43:34Z

Use this configuration if you're using MacOS and Serverless Framework

Reference: AWS Lambda Docs
custom:
  pythonRequirements:
      useStaticCache: false
      useDownloadCache: false
      pipCmdExtraArgs:
        - "--platform manylinux2014_x86_64"
        - "--implementation cp" 
        - "--python-version 3.11"
        - "--only-binary=:all:"
        - "--upgrade"
The big issue is that if you don't set useStaticCache: false and useDownloadCache: false, Serverless goes ahead and grabs MacOS cache files by default. So, when you deploy, you end up with binaries tailored for darwin instead of x86_64. Look at the pic below, you'll see the .so file having the .darwin tag.

This actually helped Thank you

siquick · 2024-03-25T05:20:01Z

Just noticed that Amazon has documentation on this which recommends passing the --platform flag to pip: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries
(This is the alternative to using pip in docker which I suggested above.)

This solution work out for me, thank you for this reference 😊.

fastapi==0.104.1 pydantic==2.5.2

plus

custom:
  pythonRequirements:
      useStaticCache: false
      useDownloadCache: false
      pipCmdExtraArgs:
        - "--platform manylinux2014_x86_64"
        - "--implementation cp" 
        - "--python-version 3.11"
        - "--only-binary=:all:"
        - "--upgrade"

this is what worked for me on Mac M1

farzadmf · 2024-03-28T21:33:03Z

Posting here in case anyone has any suggestions for AWS Glue 4.0

sungsong88 · 2024-04-04T08:35:57Z

Finally got it working with Amplify. Amplify (or pipenv) had apparently cached a non-native version of pydantic when I first set it up. Deleting the old lambda, and starting a fresh one worked once I had added a handcrafted lambda layer using...
 pip install --platform manylinux2014_x86_64 --target=python --implementation cp --python-version 3.10 --only-binary=:all: --upgrade openai
...on a linux box, zipping the content as per https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-native-libraries and uploading it to AWS via the lambda layer console. The lambda in Amplify was then updated to use that lambda layer (via its ARN).

Oh yeah! You nailed it! Thank you so much! it works for openai now!!!

ThatLowIQguy · 2024-04-29T19:29:56Z

I am still facing the error when dealing with Lambda. Upgrading nd downgrading runtime did not help. Any other suggestions?

jagdish1o1 · 2024-05-10T07:48:33Z

This worked for me:
https://stackoverflow.com/questions/77354288/no-module-named-pydantic-core-pydantic-core-server-for-python-serverless-on-a

quirozs added bug V2 Bug related to Pydantic V2 unconfirmed Bug not yet confirmed as valid/applicable labels Jul 10, 2023

pydantic-hooky bot assigned davidhewitt Jul 10, 2023

quirozs closed this as completed Jul 10, 2023

No module named 'pydantic_core._pydantic_core' using AWS Lambdas #6557

No module named 'pydantic_core._pydantic_core' using AWS Lambdas #6557

Comments

quirozs commented Jul 10, 2023 • edited by samuelcolvin

Initial Checks

Description

Example Code

Python, Pydantic & OS Version

johnbley commented Jul 10, 2023 • edited

samuelcolvin commented Jul 10, 2023

samuelcolvin commented Jul 10, 2023

johnbley commented Jul 10, 2023 • edited

samuelcolvin commented Jul 10, 2023

quirozs commented Jul 10, 2023

johnbley commented Jul 10, 2023 • edited

samuelcolvin commented Jul 10, 2023

quirozs commented Jul 10, 2023

samuelcolvin commented Jul 10, 2023

quirozs commented Jul 10, 2023

davidhewitt commented Jul 10, 2023

johnbley commented Jul 10, 2023 • edited

oriori1703 commented Jul 11, 2023 • edited

davidhewitt commented Jul 11, 2023

johnbley commented Jul 11, 2023

Aarif1430 commented Jul 13, 2023

Esquive commented Jul 18, 2023 • edited

davidhewitt commented Jul 18, 2023

Esquive commented Jul 18, 2023 • edited

davidhewitt commented Jul 19, 2023 • edited

Esquive commented Jul 19, 2023

25thbamofthetower commented Jul 20, 2023 • edited

davidhewitt commented Jul 20, 2023

paalvibe commented Jul 21, 2023

25thbamofthetower commented Jul 22, 2023 • edited

lober-io commented Dec 20, 2023

mohit2152sharma commented Dec 23, 2023

daniellersisk commented Jan 3, 2024

ResearcherSara commented Jan 11, 2024

pierluigi-failla commented Jan 11, 2024

lober-io commented Jan 11, 2024

Vanshikashah318 commented Jan 18, 2024

mohit2152sharma commented Jan 18, 2024

Vanshikashah318 commented Jan 18, 2024

NateLeeP commented Jan 24, 2024

intoxicated commented Feb 6, 2024 • edited

intoxicated commented Feb 6, 2024

intoxicated commented Feb 6, 2024

Inthuson commented Feb 6, 2024

sarflux commented Feb 9, 2024

peterrauscher commented Feb 9, 2024

bruceweir commented Feb 11, 2024

peterrauscher commented Feb 11, 2024

bruceweir commented Feb 11, 2024 • edited

andybbruno commented Feb 16, 2024

ianbbqzy commented Feb 27, 2024

davidhewitt commented Feb 27, 2024 • edited

MohammedAdain commented Mar 9, 2024 • edited

itsvaibhav01 commented Mar 12, 2024

shashidhar100 commented Mar 15, 2024

siquick commented Mar 25, 2024 • edited

farzadmf commented Mar 28, 2024

sungsong88 commented Apr 4, 2024

ThatLowIQguy commented Apr 29, 2024

jagdish1o1 commented May 10, 2024

quirozs commented Jul 10, 2023 •

edited by samuelcolvin

johnbley commented Jul 10, 2023 •

edited

johnbley commented Jul 10, 2023 •

edited

johnbley commented Jul 10, 2023 •

edited

johnbley commented Jul 10, 2023 •

edited

oriori1703 commented Jul 11, 2023 •

edited

Esquive commented Jul 18, 2023 •

edited

Esquive commented Jul 18, 2023 •

edited

davidhewitt commented Jul 19, 2023 •

edited

25thbamofthetower commented Jul 20, 2023 •

edited

25thbamofthetower commented Jul 22, 2023 •

edited

intoxicated commented Feb 6, 2024 •

edited

bruceweir commented Feb 11, 2024 •

edited

davidhewitt commented Feb 27, 2024 •

edited

MohammedAdain commented Mar 9, 2024 •

edited

siquick commented Mar 25, 2024 •

edited