Amplify can't find Python3.8 on build phase of CI/CD

[dcfranca]

** Please describe which feature you have a question about? **
Amplify Console CI/CD

** Provide additional details**
On the build phase of the default amplify configuration for deployment:
- amplifyPush --simple I'm getting an error as it can't find the Python3.8 installed

2020-04-16T08:03:36.931Z [INFO]: python3 found but version Python 3.7.4 is less than the minimum required version.
You must have python >= 3.8 installed and available on your PATH as "python3". It can be installed from https://www.python.org/downloads

The build image I'm using is the "Amazon Linux:2", which according to the documentation contains Python3.8

but even running yum list python shows only python3.7 version, no python3.8 available

I also tried to use a custom image from the Docker hub (i.e: python3.8-alpine), but they fail on the build phase without any log information.

I also tried to change my runtime requirements to Python3.8, changed it on Cloudformation files for the functions and on the Pipfile, but for some reason (I guess it requires Python3.7 for other things), it still fails with the same error.

The only way I have made it works is building python3.8 from the source, but it takes ages and it is far from an ideal solution

[dcfranca]

Any updates?

[sudeep-quantelai]

I am encountering this issue too.

[dcfranca]

@swaminator It is almost a month in pending triage state, do you have any updates on it?

[dcfranca]

1 month anniversary 🎂

[appasahebs]

Any updates on this? I am facing same issue.

[matiboy]

Same issue; just to mention one workaround (as mentioned by @DanielFranca) is to build from source: add the following to amplify.yml in the AWS console under App settings -> Build settings:

backend:
  phases:
    preBuild:
      commands:
        - export BASE_PATH=$(pwd)
        - yum install -y gcc openssl-devel bzip2-devel libffi-devel python3.8-pip
        - cd /opt && wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
        - cd /opt && tar xzf Python-3.8.2.tgz 
        - cd /opt/Python-3.8.2 && ./configure --enable-optimizations
        - cd /opt/Python-3.8.2 && make altinstall
        - pip3.8 install --user pipenv
        - ln -fs /usr/local/bin/python3.8 /usr/bin/python3
        - ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
        - cd $BASE_PATH

It would be nice if Amplify's build used the same features/options as CodeBuild's one (which includes specifying the runtime versions of Python/Java/etc)

[ravibanks54]

Any updates on this? It's driving up my build time and costs un-necessarily.

[dcfranca]

Please, post some updates regarding this issue, it is really annoying and it is slowing down the development workflow by large

@swaminator

[roeemz]

Same here, I am assuming that starting from some version Python 3.8 is a requirement, haven't investigated further.

[roeemz]

Is there a way to make the Amplify CLI work with Python 3.7 for now? a verified working version or some tweaking on its files?

I'll update if I find where the dependency comes from and how to temporarily fix it for now. (installing Python 3.8 manually consumes too much time...), just need to find the time to get to it, hopefully there will be a fix before then.

[samputer]

Another sufferer here! Would really appreciate an update on this please guys?

I suppose we could use a custom build image for this, but it seems unnecessarily complex.

[lorengordon]

Using a custom image does indeed work. Tested using nikolaik/python-nodejs:python3.8-nodejs10. Much faster than installing python3.8 in a pre-build step!

• https://hub.docker.com/r/nikolaik/python-nodejs/

[shawnmmatthews]

Any updates on this? It looks like 3.8 should be supported, but I had to add the prebuild steps mentioned by @matiboy to get the backend to build. Which is rather frustrating and slow.

[dcfranca]

@shawnmmatthews I think the best solution so far is to use a custom image like mentioned on @lorengordon post
I'm using the same image and it is working fine: https://hub.docker.com/r/nikolaik/python-nodejs/

[shawnmmatthews]

@dcfranca is there a recommended method for using a custom image? Or should I be looking into something like this https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html. I see a dropdown from the Lambda console for custom, but not somewhere to specify which image to use. Thanks for reaching out.

[dcfranca]

@shawnmmatthews You can do it from the Amplify Console.
Go to the Amplify Console, open the menu on the left, Click on "Build Settings", scroll down until you see "Build Image Settings", on the dropdown select Custom, then enter the image name on the field just below it

[shawnmmatthews]

Awesome. I assumed I was just missing the setting somewhere. Thank you so much @dcfranca!

[pchevallier]

cf: #595 (comment)
The only issue I came across using this Python3.8 support patch, is that my aws-exports.js was still not generated in my CI/CD environment.

To workaround this issue, I added the original command amplifhPush --simple at the end of the build section and my export file is now generated with the proper values.

backend:
  phases:
    build:
      commands:
        - export BASE_PATH=$(pwd)
        - yum install -y gcc openssl-devel bzip2-devel libffi-devel python3.8-pip
        - cd /opt && wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
        - cd /opt && tar xzf Python-3.8.2.tgz 
        - cd /opt/Python-3.8.2 && ./configure --enable-optimizations
        - cd /opt/Python-3.8.2 && make altinstall
        - pip3.8 install --user pipenv
        - ln -fs /usr/local/bin/python3.8 /usr/bin/python3
        - ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
        - cd $BASE_PATH
        - amplifyPush --simple

[christophebeling]

I'm astonished that there is still no fix for this. Pretty disappointing.

[roeemz]

@christophebeling , its pretty crazy that practically Amplify Console is broken and this is not getting a fix...
Even if this fix requires some cross-teams work.

[siegerts]

@christophebeling @roeemz , the build has been working fine utilizing the solution mentioned above (i.e. swapping the build image). This can be done with the https://hub.docker.com/r/nikolaik/python-nodejs/ or your own.

[lorengordon]

Any maintainers tracking or working this issue? While it is easy enough to workaround the problem using the custom image when the source is GitHub, it's more of a pain when the source is CodeCommit. This is because the "cloning repository" step attempts to run the aws command to setup the credential-helper, but the custom image does not have the aws-cli installed. So we're back to either installing python3.8 in the preBuild phase, or having to manage builds of custom images ourselves. It's sad when AWS services don't really support use of AWS services. 😢

2020-09-14T16:17:53.246Z [INFO]: aws codecommit credential-helper $@ get: 1: aws codecommit credential-helper $@ get: aws: not found
2020-09-14T16:17:53.246Z [INFO]: fatal: could not read Username for 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/.....': No such device or address
2020-09-14T16:17:53.247Z [ERROR]: !!! Unable to clone repository.

[armonge]

After the new image update i got my python builds working with

alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 10

python --version
python2 --version
python3 --version

That produces the following output

2020-12-10T10:52:32.860Z [WARNING]: + alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 10
2020-12-10T10:52:32.862Z [WARNING]: + python --version
2020-12-10T10:52:32.863Z [WARNING]: Python 2.7.18
                                    + python2 --version
2020-12-10T10:52:32.865Z [WARNING]: Python 2.7.18
2020-12-10T10:52:32.865Z [WARNING]: + python3 --version
2020-12-10T10:52:33.007Z [INFO]: Python 3.8.0

[litwicki]

@andre-scheffer is #595 (comment) an option for you?

[andre-scheffer]

@litwicki I already deploy with the installation of Python 3.8 via script. But you said in your previous post that Amplify is using the current Python version and that we should contact you if this is not the case. To me it still says that Python 3.7.9 is used if I deploy without an installation script.

[abhi7cr]

@andre-scheffer the latest build image has both python3 (python 3.7.9) and python3.8 (python 3.8) installed to be backwards compatible . Could you try running under the python3.8 cli command. You can check the exact version of python 3.8 by running python3.8 --version

[SJI-ParthaSarathi]

This saved my day! @pchevallier brother you are awesome! <3

[wvidana]

The new installation is a bit messy: all python binaries are under /usr/bin but the new python 3.8 is on /usr/local/bin, which took me several tries with convinations of update-alternatives and pip install to get it right. Also /usr/bin/pip3 is not a link but the binary from the 3.7 installation, so it can't be override with update-alternatives...

If this is the definitive solution then it should be clearly stated somewhere, so the people using python 3.8 functions can setup their amplify.yml correctly.

These lines gave me a succesfull backend deploy using the default image provided by AWS

version: 1
backend:
  phases:
    build:
      commands:
        - update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 11
        - /usr/local/bin/pip3.8 install --user pipenv
        - amplifyPush --simple

[litwicki]

@wvidana Sorry for the delayed response, but thank you for your feedback. We're going to review this and will follow up.

[SJI-ParthaSarathi]

@jake This is really frustrating that Amplify can't find python3 environment path on Windows 10 even if it is added! it only happens when I run Amplify push command :-(

…

On Thu, Jan 21, 2021 at 12:51 PM Jake Litwicki ***@***.***> wrote: @wvidana <https://github.com/wvidana> Sorry for the delayed response, but thank you for your feedback. We're going to review this and will follow up. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#595 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMO6G7KNL7GU6ZFUN6J5X4LS27FHRANCNFSM4MJLW3IQ> .

-- Partha Sarathi Kundu Software Development Engineer at SJ Innovation LLC <http://sjinnovation.com/> *Where employee happiness generates client success* Get me in touch on LinkedIn <https://www.linkedin.com/in/partha-sarathi-kundu/> Follow SJ Innovation on Facebook <https://www.facebook.com/sjinnovation/> or Twitter <https://twitter.com/sjinnovation>

[abhi7cr]

we have updated our FAQ with the commands posted by @wvidana, and will update our docs shortly. Please let us know if you are still facing issues.

[michaelbrewer]

we have updated our FAQ with the commands posted by @wvidana, and will update our docs shortly. Please let us know if you are still facing issues.

Looking forward to this being the default behavior

[glynjackson]

March 17th 2021 and this is still an issue! So frustrating, I really want to like Amplify this should be default behavior by now, right?

The workaround works for me by modifying the build script to create symlink to link python3 to python3.8. Thank you to @wvidana who posted this, saved me a lot of time

[swaminator]

Please refer to: https://github.com/aws-amplify/amplify-console/blob/master/FAQ.md#how-do-i-run-amplify-functions-with-python-runtime

[fr-an-k]

Still no fix.
if Amplify offers python lambda functions, it should support python without every customer running into problems...
Amplify is too incomplete and buggy in every way and should be marked as a beta product.

[dcfranca]

@frank-unovica I feel your pain... it has been quite frustrating for me to set up an application with Amplify... I'm not even in 100% production environment yet, and I end up on so many issues with Amplify, sometimes the only solution was to rebuild the whole environment (which is not an option if you are in production), and I know there are good workers there trying to help, but it doesn't seem to be enough for the amount of requests, and often I get myself waiting weeks, months or even years for a solution.
Last months I have been seriously considering rebuild my backend on a different tech stack and more vendor-agnostic, It will be a lot of work tho.

I'm afraid that Amplify is not production ready, I agree it should be marked as a preview product and not encouraged to go for production.

[fr-an-k]

The developers here are great people, but the Amplify approach is a complete disaster and belongs in the bin.
Switch to AWS CDK - it's so easy. I finally found what I'm looking for but it's not even listed in AWS services..
I had to throw away and rebuild our application as well due to DataStore, AppSync, federated sessions, automatic build, mocking, etc. not living up to promise, no SES or customizable resources, poor performance, undocumented undebuggable black box errors behind the scenes, and lack of help from support even though we pay hefty bills.

[dcfranca]

@frank-unovica are you still using AppSync? Have you followed some documentation or guide to move away from Amplify?

[fr-an-k]

Documentation? Guide? Welcome to disaster.

[michaelbrewer]

Yep @frank-unovica the docs for this should be on the main amplify site, and not hidden in a FAQ section on github

Basically this is the fix to ensure python 3.8 is the preferred version.

version: 1
backend:
  phases:
    build:
      commands:
        - update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 11
        - /usr/local/bin/pip3.8 install --user pipenv
        - amplifyPush --simple

[michaelangeloio]

The new installation is a bit messy: all python binaries are under /usr/bin but the new python 3.8 is on /usr/local/bin, which took me several tries with convinations of update-alternatives and pip install to get it right. Also /usr/bin/pip3 is not a link but the binary from the 3.7 installation, so it can't be override with update-alternatives...

If this is the definitive solution then it should be clearly stated somewhere, so the people using python 3.8 functions can setup their amplify.yml correctly.

These lines gave me a succesfull backend deploy using the default image provided by AWS

version: 1
backend:
  phases:
    build:
      commands:
        - update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.8 11
        - /usr/local/bin/pip3.8 install --user pipenv
        - amplifyPush --simple

Thank you so much.

[swirle13]

Is there a reason that python3.9 doesn't appear anywhere in /usr/bin/ and /usr/local/bin and a max of 3.8? I'm using 3.9 locally and lambda runtimes support 3.9 and I'm using features from 3.9, like type subscripting, that I'd like to be able to use 3.9 when running my amplify deploy.

I can't just switch out the 8 for a 9 in the above commands, like update-alternatives --install /usr/bin/python3 python3 /usr/local/bin/python3.9 11 as it doesn't exist anywhere in the build image.

Is there a reason the image doesn't support python3.9?

[jgroom33]

The default Docker image (as of the date of this post) contains:

• python 3.7
• python 3.8.0
• python 3.9.0

If those versions are sufficient, symlink like this:

version: 1
backend:
  phases:
    build:
      commands:
        - ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
        - ln -fs /usr/local/bin/python3.8 /usr/bin/python3
        - pip3 install --user pipenv
        - amplifyPush --simple

[Apfelraeuber]

I ran into the same issue and tried different options to modify the built script. The solution provided by @jgroom33 was the most efficient solution! Thanks a lot. This issue kept me busy for quite some time and for me it was completly intransparent why the building process fails...

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.