Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Heroku-20 #12

Closed
edmorley opened this issue Oct 29, 2020 · 4 comments · Fixed by #16
Closed

Add support for Heroku-20 #12

edmorley opened this issue Oct 29, 2020 · 4 comments · Fixed by #16
Assignees

Comments

@edmorley
Copy link
Member

Building an app using this buildpack on Heroku-20 resulted in this output (the build itself didn't fail; for which I'll file a separate issue):

    -----> Installing AWS CLI
           Running cmd: /usr/bin/python -m venv /app/.awscli
           Traceback (most recent call last):
             File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 223, in <module>
               main()
             File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 199, in main
               create_install_structure(working_dir, opts.install_dir)
             File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 77, in create_install_structure
               create_virtualenv(location=install_dir, working_dir=working_dir)
             File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 84, in _create_virtualenv_internal
               run('%s -m venv %s' % (sys.executable, location))
             File "/tmp/build_f2bbe920_/.awscli/awscli-bundle/install", line 56, in run
               raise BadRCError("Bad rc (%s) for cmd '%s': %s" % (
           __main__.BadRCError: Bad rc (1) for cmd '/usr/bin/python -m venv /app/.awscli': The virtual environment was not created successfully because ensurepip is not
           available.  On Debian/Ubuntu systems, you need to install the python3-venv
           package using the following command.
           
               apt-get install python3-venv
     
           You may need to use sudo with that command.  After installing the python3-venv
           package, recreate your virtual environment.
     
           Failing command: ['/app/.awscli/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

This is because on Heroku-20, python is now Python 3, not Python 2, and because under Python 3.7+ AWS CLI v1 tries to use venv rather than the virtualenv package:

# From the install script in https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
if GTE_PY37:
    create_virtualenv = _create_virtualenv_internal
else:
    create_virtualenv = _create_virtualenv_external

Looking at the latest AWS CLI docs, they now tell users to install AWS CLI v2 instead:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

Although that version only supports Python 3.7+, so would only work on Heroku-18+, plus has other breaking changes:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html

It seems like our options are to:

  1. Adjust the heroku-20 stack image to include the python3-venv package (which isn't a package we've previously made available)
  2. Switch this buildpack to using AWS CLI v2 for Heroku-20 only
  3. Switch this buildpack to using AWS CLI v2 for Heroku-18+

@dzuelke @dmathieu Thoughts? (Context: This blocks W-8287887).

@dzuelke
Copy link

dzuelke commented Nov 3, 2020

Option 2 (make it heroku-20+ only) seems best to me. That way a behavior change is tied to a stack update.

@edmorley
Copy link
Member Author

edmorley commented Nov 9, 2020

Option 2 (make it heroku-20+ only) seems best to me. That way a behavior change is tied to a stack update.

So mulling over this some more - I think this option is the best way to roll out the change initially, however long term we should backport the change to older stacks. The reason for this is that we hang too many breaking changes on the stack change, which then only ends up making stack upgrades harder for users, when they have to deal with half a dozen breaking changes at once.

Also for this buildpack usage is extremely low.

@edmorley
Copy link
Member Author

edmorley commented Nov 9, 2020

There are actually fewer breaking changes than I expected:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html

@cesarizu
Copy link

cesarizu commented Dec 1, 2020

I just pushed a PR that updates awscli to v2 and fixes this issue: #15

edmorley added a commit that referenced this issue Dec 1, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
edmorley added a commit that referenced this issue Dec 1, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
edmorley added a commit that referenced this issue Dec 1, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
@edmorley edmorley self-assigned this Dec 1, 2020
edmorley added a commit that referenced this issue Dec 1, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
edmorley added a commit that referenced this issue Dec 2, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
edmorley added a commit that referenced this issue Dec 2, 2020
Updates from the deprecated v1 AWS CLI to v2:
https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html
https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst

This fixes compatibility with Heroku-20.

In addition:
* the file copying approach to working around non-relocatability
   has been replaced with a symlink approach for faster build times.
* this approach also means using an `export` file is now viable,
   so one has been created allowing later buildpacks to use the CLI
   and not just runtime use-cases.
* the buildpack now outputs the installed version to the log, for
  easier debugging.

Fixes #7.
Fixes #12.
Closes W-8520499.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants