New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud-init is failing due to urllib3 dependency #3678

Closed
aabhassharma opened this Issue Oct 25, 2018 · 11 comments

Comments

Projects
None yet
10 participants
@aabhassharma

aabhassharma commented Oct 25, 2018

Seeing this issue with awscli 1.16.41, which does seem to pin boto to a newer version, which in turn seems to pick the correct urllib3 (which is where the ordered_dict problem starts), but it's still broken.
To make some sense of what I mean above:

installed version of awscli: 1.16.41
relevant commit line in awscli: 5a42121#diff-2eeaed663bd0d25b7e608891384b7298R26
relevant commit in boto: boto/botocore@c4d3b1a

Error from system log:

  File "/usr/bin/cloud-init", line 43, in <module>
    from cloudinit import netinfo
  File "/usr/lib/python2.7/dist-packages/cloudinit/netinfo.py", line 23, in <module>
    import cloudinit.util as util
  File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 59, in <module>
    from cloudinit import url_helper
  File "/usr/lib/python2.7/dist-packages/cloudinit/url_helper.py", line 27, in <module>
    import requests
  File "/usr/lib/python2.7/dist-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/dist-packages/requests/utils.py", line 25, in <module>
    from .compat import parse_http_list as _parse_list_header
  File "/usr/lib/python2.7/dist-packages/requests/compat.py", line 92, in <module>
    from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict```

Relevant discussion from urllib3: 
https://github.com/urllib3/urllib3/issues/1456
@ctud

This comment has been minimized.

ctud commented Oct 25, 2018

I have also seen this issue. Installing the latest aws-cli on an image is preventing new AMIs based on that image from booting due to the above issue with urllib3.

@mflopez79

This comment has been minimized.

mflopez79 commented Oct 25, 2018

I'm facing the same issue on Ubuntu 14.04 (AWS EC2), but it's reproducible on Vagrant. Boot a clean machine and cloud-init works (check with cloud-init init). Get awscli installed with pip and cloud-init shows the same error that @aabhassharma reported when cloud-init init is executed

@mflopez79

This comment has been minimized.

mflopez79 commented Oct 25, 2018

Did some more checks, version 1.16.38 is the last one that doesn't break cloud-init. Starting from version 1.16.39 it breaks cloud-init imports.
What I did as a workaround was install version 1.16.38 as

pip install awscli==1.16.38
@joguSD

This comment has been minimized.

Contributor

joguSD commented Oct 29, 2018

This is an issue with how Python packages are installed and upstream in urllib3/requests.
AWS CLI v1.16.39 and greater bumped the version range for urllib3, so pip is pulling in the latest version of urllib3 which is incompatible with the version of requests that cloud-init is using.

You can use newer versions of the CLI, you'll just need to help pip by specifying that you want an older version of urllib3==1.23. You could also try pulling a newer version of requests, which supports the latest version of urllib3.

See: urllib3/urllib3#1456

@ciandro

This comment has been minimized.

ciandro commented Oct 31, 2018

I have forced a requests update to v2.20 which takes care of urllib3 1.24 and now all my automation is working.

@no-response

This comment has been minimized.

no-response bot commented Nov 5, 2018

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

@richid

This comment has been minimized.

richid commented Nov 7, 2018

This issue should be reopened. I've spent many hours trying to track this down and I am completely horrified that installing a new version of awscli is responsible for breaking my instance launches.

@woqer

This comment has been minimized.

woqer commented Nov 8, 2018

This is still going on. I had to specifically pip upgrade requests in my packer builds to avoid cloud-init failure

@joguSD

This comment has been minimized.

Contributor

joguSD commented Nov 8, 2018

@richid Unfortunately, there's nothing we can do to fix this for you. This is not an issue with the awscli, but an issue with installing any Python package globally that allows a newer version of urllib3 that conflicts with cloud-init. In general, installing Python packages globally into a shared environment can be problematic.

There's three options to get around this issue:

  1. Carefully curate your global Python environment to ensure you're installing compatible versions of all your packages. In this case, installing an older version of urllib3 or a newer version of requests.
  2. Properly isolate Python packages into virtual environments so tools can have their own set of isolated dependencies.
  3. Use the bundled installer to install the AWS CLI which will handle setting up an isolated virtual environment to avoid conflicts like these.
@patrobinson

This comment has been minimized.

patrobinson commented Nov 12, 2018

This is one of the many reasons why awscli should just be a statically compiled binary. It's typically required to be installed on a system with various other requirements and as such is easily broken by requirements from other dependencies.

@mipearson

This comment has been minimized.

mipearson commented Nov 14, 2018

@joguSD Is this official AWS advice, specifically option 3? This just bit us (twice) and we're preparing a general advice for other teams so that they can avoid their instances not booting.

aberoham added a commit to gravitational/teleport that referenced this issue Nov 15, 2018

aberoham added a commit to gravitational/teleport that referenced this issue Nov 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment