tox -e ... on charms fails with ResolutionImpossible

[lourot]

Since yesterday (2021-07-20) the pip resolver seems to behave differently and fails on following conflicts:

The conflict is caused by:
    The user requested zaza 0.0.2.dev1 (from git+https://github.com/openstack-charmers/zaza.git#egg=zaza;python_version>='3.0')
    zaza-openstack 0.0.1.dev1 depends on zaza 0.0.2.dev1 (from git+https://github.com/openstack-charmers/zaza.git#egg=zaza)

The conflict is caused by:
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.8.3 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.8.2 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.8.1 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.8 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.7 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.6 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.5 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.3 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.2 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.7.0 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.6.1 depends on jsonschema<=2.5.1
    tempest 27.0.1.dev70 depends on jsonschema>=3.2.0
    charm-tools 2.6.0 depends on jsonschema<=2.5.1
    zaza 0.0.2.dev1 depends on PyYAML
    zaza-openstack 0.0.1.dev1 depends on PyYAML
    tempest 27.0.1.dev70 depends on PyYAML>=3.12
    charm-tools 2.5.1 depends on pyyaml==3.11
    zaza 0.0.2.dev1 depends on PyYAML
    zaza-openstack 0.0.1.dev1 depends on PyYAML
    tempest 27.0.1.dev70 depends on PyYAML>=3.12
    charm-tools 2.5.0 depends on pyyaml==3.11
    zaza 0.0.2.dev1 depends on PyYAML
    zaza-openstack 0.0.1.dev1 depends on PyYAML
    tempest 27.0.1.dev70 depends on PyYAML>=3.12
    charm-tools 2.4.5 depends on pyyaml==3.11
    zaza 0.0.2.dev1 depends on PyYAML
    zaza-openstack 0.0.1.dev1 depends on PyYAML
    tempest 27.0.1.dev70 depends on PyYAML>=3.12
    charm-tools 2.4.4 depends on pyyaml==3.11

The first conflict is easy to solve by removing ;python_version>='3.0' in test-requirements.txt

The second conflict is legit: we want to install the latest tempest (we need it as a binary), but at the same time we want to install charm-tools, which pins older libraries to keep py3.5 support. The solution: we don't actually need to install charm-tools and tempest in the same venvs. charm-tools is needed for pep8 whereas tempest is needed for functional tests (Zaza runs). This has been solved properly in this repo: https://github.com/openstack-charmers/charmed-openstack-tester/blob/master/tox.ini#L29

This is hitting us in all classic charms. So far seen on:

• ceilometer-agent
• glance
• neutron-api (e.g. on this review: https://review.opendev.org/c/openstack/charm-neutron-api/+/801513)

I'm working on a fix.

[johnlettman]

I can confirm first and second conflict in ceph-osd.

[lourot]

Validating a fix here: https://review.opendev.org/c/openstack/charm-glance/+/801635

If this turns green, we'll have to do this change in all classic charms

[johnlettman]

Attempted identical changes on ceph-osd.
tox -e pep8 throws "ERROR: InvocationError for command could not find executable flake8".

Adding flake8 below charm-tools resolves this pip error.

diff --git a/test-requirements.txt b/test-requirements.txt
index 9aea716..57e54db 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -8,7 +8,6 @@
 #       all of its own requirements and if it doesn't, fix it there.
 #
 setuptools<50.0.0  # https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c44b7c85
-charm-tools>=2.4.4
 
 # Workaround until https://github.com/juju/charm-tools/pull/589 gets
 # published
@@ -42,7 +41,7 @@ oslo.utils<=3.41.0;python_version<'3.6'
 
 coverage>=4.5.2
 pyudev              # for ceph-* charm unit tests (need to fix the ceph-* charm unit tests/mocking)
-git+https://github.com/openstack-charmers/zaza.git#egg=zaza;python_version>='3.0'
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
 git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
 
 # Needed for charm-glance:
diff --git a/tox.ini b/tox.ini
index ab9593f..105a620 100644
--- a/tox.ini
+++ b/tox.ini
@@ -65,8 +65,7 @@ deps = -r{toxinidir}/requirements.txt
 
 [testenv:pep8]
 basepython = python3
-deps = -r{toxinidir}/requirements.txt
-       -r{toxinidir}/test-requirements.txt
+deps = charm-tools
 commands = flake8 {posargs} hooks unit_tests tests actions lib files
            charm-proof

[johnlettman]

These changes cause a dependency conflict in py36, py37, py38, and cover (may also break func):

ERROR: Cannot install -r /home/zuul/src/opendev.org/openstack/charm-glance/test-requirements.txt (line 47) and pbr<1.9.0 and >=1.8.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pbr<1.9.0 and >=1.8.0
    tempest 27.0.1.dev70 depends on pbr!=2.1.0 and >=2.0.0

Possible fix

I changed the version pinning in requirements.txt for pbr to match that requested from tempest. This change appears to resolve the conflict in my local invocation of those tests:

py36

$ tox -e py36 --notest
[SNIP]
  py36: skipped tests
  congratulations :)

py37

$ tox -e py37 --notest
[SNIP]
  py37: skipped tests
  congratulations :)

py38

$ tox -e py38 --notest
[SNIP]
  py38: skipped tests
  congratulations :)

cover

$ tox -e cover
[SNIP]
  cover: commands succeeded
  congratulations :)

Current diff

diff --git a/requirements.txt b/requirements.txt
index 360ecba..e1c42ca 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -7,7 +7,8 @@
 #       requirements.  They are intertwined.  Also, Zaza itself should specify
 #       all of its own requirements and if it doesn't, fix it there.
 #
-pbr>=1.8.0,<1.9.0
+pbr>=2.0.0,!=2.1.0
+
 simplejson>=2.2.0
 netifaces>=0.10.4
 
diff --git a/test-requirements.txt b/test-requirements.txt
index 9aea716..57e54db 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -8,7 +8,6 @@
 #       all of its own requirements and if it doesn't, fix it there.
 #
 setuptools<50.0.0  # https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c44b7c85
-charm-tools>=2.4.4
 
 # Workaround until https://github.com/juju/charm-tools/pull/589 gets
 # published
@@ -42,7 +41,7 @@ oslo.utils<=3.41.0;python_version<'3.6'
 
 coverage>=4.5.2
 pyudev              # for ceph-* charm unit tests (need to fix the ceph-* charm unit tests/mocking)
-git+https://github.com/openstack-charmers/zaza.git#egg=zaza;python_version>='3.0'
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
 git+https://github.com/openstack-charmers/zaza-openstack-tests.git#egg=zaza.openstack
 
 # Needed for charm-glance:
diff --git a/tox.ini b/tox.ini
index ab9593f..9ba3f9f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -65,8 +65,8 @@ deps = -r{toxinidir}/requirements.txt
 
 [testenv:pep8]
 basepython = python3
-deps = -r{toxinidir}/requirements.txt
-       -r{toxinidir}/test-requirements.txt
+deps = flake8==3.9.2
+       charm-tools==2.8.3
 commands = flake8 {posargs} hooks unit_tests tests actions lib files
            charm-proof
 

[lourot]

Thanks for the analysis @jlettman ! Updating https://review.opendev.org/c/openstack/charm-glance/+/801635 accordingly

[lourot]

Note: this is not hitting source charms (e.g. it didn't hit this gate: https://review.opendev.org/c/openstack/charm-vault/+/801042 ), so only classic charms seem to be affected

[lourot]

Charm review batch: https://review.opendev.org/q/topic:fix-tox-pip-resolutionimpossible