charm-build -w doesn't actually do overrides anymore (due to duplicate module checking code that has recently changed)

[ajkavanagh]

Checklist

• Confirmed this is an issue with charm-tools, not charmstore-client
• [X ] Provide versions of tools used
• [X ] Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm and charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact:   https://discourse.juju.is/c/charming
license:   unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     latest/stable
refresh-date: today at 16:43 BST
channels:
  latest/stable:    2.7.5 2020-06-26 (481) 116MB classic
  latest/candidate: 2.7.5 2020-06-26 (481) 116MB classic
  latest/beta:      2.7.5 2020-06-26 (481) 116MB classic
  latest/edge:      2.7.5 2020-06-25 (481) 116MB classic
installed:          2.7.5            (481) 116MB classic

I am using: (Your operating system) Bionic

Issue/Feature

"charm build" with competing module versions fails due to recent changes to deal with overlapping/competing versions. However, it looks like the -w option to provide an override doesn't work any more.

I expect/expected the following

I expected the -w option to override and use those version. Instead:

What I got

$ charm build --log-level DEBUG -o build/ src -w wheelhouse.txt

...
charmtools.build.tactics: Processing wheelhouse:
charmtools.build.tactics:   # layer:basic
charmtools.build.tactics:   # pip is pinned to <19.0 to avoid https://github.com/pypa/pip/issues/6164
charmtools.build.tactics:   # even with installing setuptools before upgrading pip ends up with pip seeing
charmtools.build.tactics:   # the older setuptools at the system level if include_system_packages is true
charmtools.build.tactics:   pip>=18.1,<19.0
charmtools.build.tactics:   # pin Jinja2 and PyYAML to the last versions supporting python 3.4 for trusty
charmtools.build.tactics:   Jinja2<=2.10.1
charmtools.build.tactics:   PyYAML<=5.2
charmtools.build.tactics:   setuptools<42
charmtools.build.tactics:   setuptools-scm<=1.17.0
charmtools.build.tactics:   charmhelpers>=0.4.0,<1.0.0
charmtools.build.tactics:   charms.reactive>=0.1.0,<2.0.0
charmtools.build.tactics:   wheel<0.34
charmtools.build.tactics:   # pin netaddr to avoid pulling importlib-resources
charmtools.build.tactics:   netaddr<=0.7.19
charmtools.build.tactics:
charmtools.build.tactics:   # layer:openstack
charmtools.build.tactics:   # charmhelpers.contrib.openstack.cert_utils charms with certificates relation
charmtools.build.tactics:   dnspython
charmtools.build.tactics:   # charmhelpers.contrib.network all
charmtools.build.tactics:   netifaces>=0.10.4
charmtools.build.tactics:   # charmhelpers.contrib.openstack.context all
charmtools.build.tactics:   psutil
charmtools.build.tactics:   # charms_openstack - pull from github until first release
charmtools.build.tactics:   git+https://github.com/openstack/charms.openstack.git#egg=charms.openstack
charmtools.build.tactics:
charmtools.build.tactics:   # manila
charmtools.build.tactics:
charmtools.build.tactics:   git+https://opendev.org/openstack/charms.openstack.git@stable/20.05#egg=charms.openstack
charmtools.build.tactics:
charmtools.build.tactics:   git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers
charmtools.build.tactics:
utils: b"ERROR: Double requirement given: charmhelpers from git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 31)) (already in charmhelpers<1.0.0,>=0.4.0 (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 11)), name='charmhelpers')\n"
utils: process: bash -c . /tmp/tmp4cfy3aj8/bin/activate ; pip3 download --no-binary :all: -d /tmp/tmpwkfqf3jx -r /tmp/tmppicybgp7/wheelhouse.txt (1)

...

charmtools.build.errors.BuildError: ERROR: Double requirement given: charmhelpers from git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 31)) (already in charmhelpers<1.0.0,>=0.4.0 (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 11)), name='charmhelpers')

build: ERROR: Double requirement given: charmhelpers from git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 31)) (already in charmhelpers<1.0.0,>=0.4.0 (from -r /tmp/tmppicybgp7/wheelhouse.txt (line 11)), name='charmhelpers')

wheelhouse.txt:

$ cat wheelhouse.txt

git+https://opendev.org/openstack/charms.openstack.git@stable/20.05#egg=charms.openstack

git+https://github.com/juju/charm-helpers.git@stable/20.05#egg=charmhelpers

[johnsca]

I think this isn't a general problem with --wheelhouse-overrides and the new combined wheelhouse processing, but rather an issue with it ignoring git+ style requirements due to them not being supported by pkg_resources.Requirement.parse. There are a couple alternative options for parsing the requirements: requirements-parser, requirementslib, or I also notice that pbr has a requirement parser that seems like it would handle those lines as well.

[thedac]

Cory, this is a fairly high priority for us, in that our stable branch all override charm helpers, charms.openstack and charms.ceph to stable branch cuts. So in effect, our stable gate is broken. Let me know what I can do to assist.

[ajkavanagh]

Thanks for this patch/fix. However, note there is a new problem related to the the changed requirements parser: #573