Add PyCryptodome to requirements_ansible.*, related #3180 #3305

skinlayers · 2019-02-25T05:22:12Z

related #3180

SUMMARY

Adds PyCryptodome to python requirements. Fixes gce.py dynamic inventory failure.
related #3180

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

inventory_import.py & gce.py

AWX VERSION

awx: 3.0.1

ADDITIONAL INFORMATION

Before:

2019-02-21 08:52:29,422 INFO     awx.main.commands.inventory_import Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/gce.py
2019-02-21 08:52:29,423 INFO     awx.main.commands.inventory_import Using VIRTUAL_ENV: /var/lib/awx/venv/ansible
2019-02-21 08:52:29,424 INFO     awx.main.commands.inventory_import Using PATH: /var/lib/awx/venv/ansible/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2019-02-21 08:52:29,424 INFO     awx.main.commands.inventory_import Using PYTHONPATH: /var/lib/awx/venv/ansible/lib/python2.7/site-packages:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/bin/awx-manage", line 11, in <module>
    load_entry_point('awx==3.0.1.0', 'console_scripts', 'awx-manage')()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/__init__.py", line 140, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 1049, in handle
    raise exc
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 949, in handle
    data = AnsibleInventoryLoader(source=source, is_custom=self.is_custom, venv_path=venv_path).load()
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 195, in load
    return self.command_to_json(base_args + ['--list'])
  File "/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/management/commands/inventory_import.py", line 178, in command_to_json
    self.method, proc.returncode, stdout, stderr))
RuntimeError: ansible-inventory failed (rc=1) with stdout:
stderr:
 [WARNING]:  * Failed to parse /var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py with script plugin: Inventory script
(/var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py) had an execution error: Traceback (most
recent call last):   File "/var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py", line 508, in <module>
GceInventory()   File "/var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py", line 170, in __init__     self.driver =
self.get_gce_driver()   File "/var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py", line 318, in get_gce_driver     gce =
get_driver(Provider.GCE)(*args, **kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/compute/drivers/gce.py", line 1795, in __init__
super(GCENodeDriver, self).__init__(user_id, key, **kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/common/base.py", line 975, in __init__     self.connection =
self.connectionCls(*args, **conn_kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/compute/drivers/gce.py", line 99, in __init__
credential_file=credential_file, **kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/common/google.py", line 767, in __init__     user_id, key,
auth_type, credential_file, scopes, **kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/common/google.py", line 653, in __init__     self.user_id,
self.key, self.scopes, **kwargs)   File
"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/common/google.py", line 486, in __init__     raise
GoogleAuthError('PyCrypto library required for '
libcloud.common.google.GoogleAuthError: 'PyCrypto library required for Service
Account Authentication.'
 [WARNING]: Unable to parse /var/lib/awx/venv/awx/lib64/python3.6/site-
packages/awx/plugins/inventory/gce.py as an inventory source
ERROR! No inventory was parsed, please check your configuration and options.

After:

2019-02-25 05:10:28,348 INFO     awx.main.commands.inventory_import Updating inventory 2: foo-gcp-project
2019-02-25 05:10:28,367 INFO     awx.main.commands.inventory_import Reading Ansible inventory source: /var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/plugins/inventory/gce.py
2019-02-25 05:10:28,368 INFO     awx.main.commands.inventory_import Using VIRTUAL_ENV: /var/lib/awx/venv/ansible
2019-02-25 05:10:28,368 INFO     awx.main.commands.inventory_import Using PATH: /var/lib/awx/venv/ansible/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/var/lib/awx/venv/awx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2019-02-25 05:10:28,368 INFO     awx.main.commands.inventory_import Using PYTHONPATH: /var/lib/awx/venv/ansible/lib/python2.7/site-packages:
2019-02-25 05:10:31,766 INFO     awx.main.commands.inventory_import Processing JSON output...
2019-02-25 05:10:31,767 INFO     awx.main.commands.inventory_import Loaded 31 groups, 12 hosts
2019-02-25 05:10:32,504 DEBUG    awx.main.models.inventory Going to update inventory computed fields, pk=2
2019-02-25 05:10:33,196 DEBUG    awx.main.models.inventory Finished updating inventory computed fields, pk=2, in 0.692 seconds
2019-02-25 05:10:33,211 INFO     awx.main.commands.inventory_import Inventory import completed for foo-gcp-project in 4.9s

Fixes issue #3180

softwarefactory-project-zuul · 2019-02-25T05:36:03Z

Build failed.

tox-awx-api-lint : SUCCESS in 2m 12s
tox-awx-ui-lint : SUCCESS in 2m 45s
tox-awx-api : FAILURE in 11m 19s
tox-awx-ui : SUCCESS in 4m 02s
tox-awx-swagger : SUCCESS in 12m 51s
tox-awx-detect-schema-change : SUCCESS in 12m 35s (non-voting)

softwarefactory-project-zuul · 2019-02-25T08:06:02Z

Build succeeded.

tox-awx-api-lint : SUCCESS in 2m 11s
tox-awx-ui-lint : SUCCESS in 2m 20s
tox-awx-api : SUCCESS in 11m 15s
tox-awx-ui : SUCCESS in 3m 48s
tox-awx-swagger : SUCCESS in 12m 15s
tox-awx-detect-schema-change : SUCCESS in 6m 11s (non-voting)

skinlayers · 2019-02-25T08:24:08Z

I may take a stab at updating some of the requirements in a controlled fashion in a separate PR. I tried to follow the instructions in requirements/README.md, but at least one of the updated packages broke my AWX test deployment.

matburt · 2019-02-25T14:06:05Z

We probably need to see if this requirement is needed for: #3266

This PR obviates any other inventory script related changes we'd make here.

matburt · 2019-02-25T14:14:47Z

What part of the script needs this? I don't run into this during normal usage.

skinlayers · 2019-02-25T17:35:16Z

@matburt google-auth does not appear to require PyCryptodome.

~$ docker run --rm -it python:3.6 bash -c 'pip list && pip install google-auth'
Package    Version
---------- -------
pip        18.1
setuptools 40.4.3
wheel      0.32.2
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting google-auth
  Downloading https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 2.2MB/s
Collecting six>=1.9.0 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 2.0MB/s
Collecting cachetools>=2.0.0 (from google-auth)
  Downloading https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa>=3.1.4->google-auth)
  Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 1.1MB/s
Installing collected packages: six, pyasn1, rsa, pyasn1-modules, cachetools, google-auth
Successfully installed cachetools-3.1.0 google-auth-1.6.3 pyasn1-0.4.5 pyasn1-modules-0.2.4 rsa-4.0 six-1.12.0

However, the open PR doesn't help me at that moment, whereas including this dependency does. ;)
All joking aside, even if this PR is not merged, at least it will serve to document a workaround anyone else who encounters this issue prior to 4.0.0/switching to inventory plugins.

Are you testing the gce.py inventory script in Kubernetes/GKE?
From the error message:

"/var/lib/awx/venv/ansible/lib/python2.7/site-
packages/libcloud/common/google.py", line 486, in __init__     raise
GoogleAuthError('PyCrypto library required for '
libcloud.common.google.GoogleAuthError: 'PyCrypto library required for Service
Account Authentication.'

The GCE dynamic inventory script requires apache-libcloud and credentials from a GCP service account. libcloud in turn, requires PyCrypto (or preferably, the newer PyCryptodome) to use a GCP service account's credentials. I've tested this bug in both 3.0.1 and the latest commits in the devel branch.

AlanCoding · 2019-02-26T16:29:15Z

Took me a while, but I got around to double-checking your work. I can confirm that I reproduced the linked issue report, and that adding this dependency resolves the issue. I only reproduced locally, running ansible-inventory on the CLI, but I have no concerns about the ability to translate that to inventory updates in AWX.

What I did:

create a python3 virtual environment
install Anisble, libcloud, google-auth
run ./contrib/inventory/gce.py, plus the additional setup, you don't want to see this, it's gory
observe the reported traceback
install PyCryptodome
run the script again
see that it runs correctly

Double checking plugin:

after those steps, ran ansible-inventory -i gcp_compute.yml --list --export
still runs

I do see the gap in testing. I know for sure that we did:

using custom virtualenvs for inventory imports
using python3 virtualenvs for job runs

To my knowledge, we never got around to testing the combination these new features - running inventory updates in python3, ping @kdelee @ryanpetrello. My interest for inventory updates with custom virtual environments was to test inventory plugins. I think python3 is important to cross-test with, but it's not a priority for me until the inventory plugins are in. We might also want regression testing (which is easily viable), before merging this. Also, I'm worried about dependency conflicts in more than just gce, so that needs testing too.

As for this PR, it's under consideration, but there are a lot of questions, so I hope you understand that we will delay for a while.

Here's the background I have on this issue:

ansible/ansible#18453

See particular comment

Please note pycrypto is waaay out of date, nearing 3 years since last release at this point. pycryptodome would be a better requirement; it's a drop-in, and its latest release was 6 days ago- it is actively developed.

This is an argument for the solution you took here. I understand, it is well-stated. However, here is what the Ansible core team did in response to that issue:

https://github.com/ansible/ansible/pull/21430/files

pycrypto >= 2.6

To double check, I did:

uninstalled pycryptodome
installed pycrypto (==2.6.1, Oct 17, 2013)
confirmed script still runs
confirmed plugin still runs

Picking one library versus another is a high-consequence decision. I will not be the one to make it. If it is decided that we use this one, I'll be happy to merge this change. If not, we still appreciate the help. Thanks!

EDIT: missed a step

AlanCoding · 2019-02-27T14:29:52Z

@skinlayers from the issue, it looks like you were running in the default ansible virtualenv. Is that right? I can't square that with my local testing, and when I test gce sources in AWX, they run successfully.

AlanCoding · 2019-02-27T14:32:02Z

If I run this in gce updates in a python3 virtualenv, then I do hit the error in the linked issue.

I just need to be clear about what we're dealing with here.

ryanpetrello · 2019-02-27T16:46:54Z

@AlanCoding if this is a matter of only installing certain packages (PyCrypto) for py3 venvs, pip does support this type of thing:

awx/requirements/requirements_ansible.txt

Line 53 in d2fa5cc

    
           enum34==1.1.6; python_version < '3' # via cryptography, knack, msrest, ovirt-engine-sdk-python

What if we did something like:

pycrypto==2.6.1; python_version > '3' # necessary for gce/libcloud to work on py3

...in requirements_ansible.txt ?

AlanCoding · 2019-02-27T17:13:59Z

requirements_ansible.txt should never be installed for python3 in our current system.

That has a lot of potential fallout, and it doesn't match the facts of the reported issues. Some of these issues mention python3, but what does that mean? Was the default venv hacked to use python3 and not python2? That's going to be a problem since we're trying to use system site packages in the default venv.

Depending on what is actually going on with that issue, it's going to be a completely different conversation. Using a python3 default venv will take work dedicated specifically to that goal, I think you might be doing that.

The inventory plugins, at least, will need to run in python3. Best thing we can do might be to start active work on testing that soon.

wenottingham · 2019-02-27T20:10:44Z

That has a lot of potential fallout, and it doesn't match the facts of the reported issues. Some of these issues mention python3, but what does that mean? Was the default venv hacked to use python3 and not python2? That's going to be a problem since we're trying to use system site packages in the default venv.

The default Ansible venv uses the default system python. On certain OSes, that is python3, on others, that is python2. It would use requirements_ansible.txt in both cases, and inherit system site-packages in both cases. (cc @ryanpetrello for a fact-check)

skinlayers · 2019-03-04T21:11:52Z

My setup:

Using existing GKE cluster.

mkdir ansible-github
cd ansible-github
git clone https://github.com/ansible/awx.git
git clone https://github.com/ansible/awx-logos.git
cp awx/installer/inventory ./my-inventory-devel

cd awx
echo 'devel' > ./VERSION
python3 -m venv venv
. ./venv/bin/activate
pip install -U pip setuptools
pip install ansible docker

cd installer
ansible-playbook -i ../../my-inventory-devel install.yml
kubectl delete ing awx-web-svc -n awx
kubectl apply -f ../../ingress-with-cert-manager.yaml

my-inventory-devel modifications:

  # dockerhub_base=ansible
  kubernetes_context=gke_foo-gcp-project_us-central1_bar-gke-cluster
  kubernetes_namespace=awx
  kubernetes_web_image=gcr.io/foo-gcp-project/awx/awx_web
  kubernetes_task_image=gcr.io/foo-gcp-project/awx/awx_task
  docker_registry=gcr.io
  docker_registry_repository=foo-gcp-project/awx
  docker_registry_username=oauth2accesstoken
  pg_password=CHANGME
  rabbitmq_password=CHANGEME_2
  rabbitmq_erlang_cookie=CHANGEME_3
  admin_password=CHANGEME_4
  create_preload_data=False
  secret_key=CHANGEME_5
  awx_official=true

ingress-with-cert-manager.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # See https://docs.cert-manager.io/en/latest/tasks/issuing-certificates/ingress-shim.html#supported-annotations
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    # add an annotation indicating the issuer to use.
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
  name: awx-test-internal-example-com-ingress
  namespace: awx
spec:
  rules:
  - host: awx.test.internal.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: awx-web-svc
          servicePort: 80
  tls: # < placing a host in the TLS config will indicate a cert should be created
  - hosts:
    - awx.test.internal.unity3d.com
    secretName: awx-test-internal-example-com # < cert-manager will store the created certificate in this secret

ryanpetrello · 2019-03-18T13:23:26Z

@skinlayers I've actually opened a PR to libcloud to replace PyCrypto usage with a more modern dependency awx already uses (https://pypi.org/project/cryptography/):

apache/libcloud#1280

If it merges and is released, #3180 will be resolved, so please feel to chime in with your support there.

ryanpetrello · 2019-03-19T20:12:05Z

FYI libcloud now no longer uses PyCrypto (apache/libcloud#1280); once a new release lands on PyPI, we'll update AWX's dependencies.

Add PyCryptodome to requirements_ansible.*

591c2f1

Fixes issue #3180

awxbot added state:needs_triage type:bug labels Feb 25, 2019

Add license for pycryptodome

d05f517

AlanCoding mentioned this pull request Feb 26, 2019

Move to using a requirements.txt to install the python packages. ansible/ansible#21430

Merged

AlanCoding mentioned this pull request Mar 14, 2019

plugins/tower.py: Use urllib.parse rather than urlparse #3416

Merged

ryanpetrello closed this Mar 18, 2019

ryanpetrello mentioned this pull request Mar 21, 2019

pin apache-libcloud to a version that doesn't use PyCrypto #3466

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PyCryptodome to requirements_ansible.*, related #3180 #3305

Add PyCryptodome to requirements_ansible.*, related #3180 #3305

skinlayers commented Feb 25, 2019 •

edited

Loading

softwarefactory-project-zuul bot commented Feb 25, 2019

softwarefactory-project-zuul bot commented Feb 25, 2019

skinlayers commented Feb 25, 2019

matburt commented Feb 25, 2019

matburt commented Feb 25, 2019

skinlayers commented Feb 25, 2019 •

edited

Loading

AlanCoding commented Feb 26, 2019 •

edited

Loading

AlanCoding commented Feb 27, 2019

AlanCoding commented Feb 27, 2019

ryanpetrello commented Feb 27, 2019

AlanCoding commented Feb 27, 2019

wenottingham commented Feb 27, 2019

skinlayers commented Mar 4, 2019 •

edited

Loading

ryanpetrello commented Mar 18, 2019 •

edited

Loading

ryanpetrello commented Mar 19, 2019

Add PyCryptodome to requirements_ansible.*, related #3180 #3305

Add PyCryptodome to requirements_ansible.*, related #3180 #3305

Conversation

skinlayers commented Feb 25, 2019 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

AWX VERSION

ADDITIONAL INFORMATION

softwarefactory-project-zuul bot commented Feb 25, 2019

softwarefactory-project-zuul bot commented Feb 25, 2019

skinlayers commented Feb 25, 2019

matburt commented Feb 25, 2019

matburt commented Feb 25, 2019

skinlayers commented Feb 25, 2019 • edited Loading

AlanCoding commented Feb 26, 2019 • edited Loading

AlanCoding commented Feb 27, 2019

AlanCoding commented Feb 27, 2019

ryanpetrello commented Feb 27, 2019

AlanCoding commented Feb 27, 2019

wenottingham commented Feb 27, 2019

skinlayers commented Mar 4, 2019 • edited Loading

ryanpetrello commented Mar 18, 2019 • edited Loading

ryanpetrello commented Mar 19, 2019

skinlayers commented Feb 25, 2019 •

edited

Loading

skinlayers commented Feb 25, 2019 •

edited

Loading

AlanCoding commented Feb 26, 2019 •

edited

Loading

skinlayers commented Mar 4, 2019 •

edited

Loading

ryanpetrello commented Mar 18, 2019 •

edited

Loading