Pre-install setuptools-scm <6 in venv

[conorsch]

Status

Ready for review

Description of Changes

Fixes #5876

Required in order to maintain support for Python 3.5, under Xenial.
The python-dateutil package pulls in whatever the latest setuptools-scm
version is, and as of setuptools_scm>=6, Python 3.5 is no longer
supported.

Testing

CI must be passing.

Deployment

Unbreaks Xenial deb builds, so critical for deployment.

[conorsch]

Marking as WIP since ideally we'd pin an explicit hash for this build-time dependency. Submitting without hash-pinning to evaluate how CI likes it.

[kushaldas]

Still no luck.

Downloading https://files.pythonhosted.org/packages/c5/39/4da7c2dbc4f023fba5fb2325febcadf0d0ce0efdc8
bd12083a0f65d20653/python-dateutil-2.7.2.tar.gz (298kB)", "  Downloading from URL https://files.pythonhosted.org/packages/c5/39/4da7c2dbc4f023fba5fb2325febcad
f0d0ce0efdc8bd12083a0f65d20653/python-dateutil-2.7.2.tar.gz#sha256=9d8074be4c993fbe4947878ce593052f71dac82932a677d49194d8ce9778002e (from https://pypi.org/sim
ple/python-dateutil/)", "  Running setup.py (path:/tmp/pip-build-knaads8k/python-dateutil/setup.py) egg_info for package python-dateutil", "    Running comman
d python setup.py egg_info", "    WARNING: The wheel package is not available.", "    Traceback (most recent call last):", "      File \"<string>\", line 1, i
n <module>", "      File \"/tmp/pip-build-knaads8k/python-dateutil/setup.py\", line 86, in <module>", "        \"test\": Unsupported", "      File \"/tmp/secu
redrop-app-code_1.9.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/setuptools/__init__.py\", line 144
, in setup", "        return distutils.core.setup(**attrs)", "      File \"/usr/lib/python3.5/distutils/core.py\", line 108, in setup", "        _setup_distri
bution = dist = klass(attrs)", "      File \"/tmp/securedrop-app-code_1.9.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-app-code/lib/pyth
on3.5/site-packages/setuptools/dist.py\", line 425, in __init__", "        k: v for k, v in attrs.items()", "      File \"/usr/lib/python3.5/distutils/dist.py
\", line 281, in __init__", "        self.finalize_options()", "      File \"/tmp/securedrop-app-code_1.9.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/ve
nvs/securedrop-app-code/lib/python3.5/site-packages/setuptools/dist.py\", line 706, in finalize_options", "        ep.load()(self)", "      File \"/tmp/secure
drop-app-code_1.9.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pkg_resources/__init__.py\", line 22
29, in load", "        return self.resolve()", "      File \"/tmp/securedrop-app-code_1.9.0~rc1+xenial_amd64/debian/securedrop-app-code/opt/venvs/securedrop-a
pp-code/lib/python3.5/site-packages/pkg_resources/__init__.py\", line 2235, in resolve", "        module = __import__(self.module_name, fromlist=['__name__'],
 level=0)", "      File \"/tmp/pip-build-knaads8k/python-dateutil/.eggs/setuptools_scm-6.0.1-py3.5.egg/setuptools_scm/__init__.py\", line 15, in <module>", " 
       from .version import format_version, meta", "      File \"/tmp/pip-build-knaads8k/python-dateutil/.eggs/setuptools_scm-6.0.1-py3.5.egg/setuptools_scm/v
ersion.py\", line 42", "        trace(f\"tag '{tag}' parsed to {result}\")", "                                              ^", "    SyntaxError: invalid synt
ax", "Cleaning up...", "

It is getting setuptools_scm==6.0.1 for dateutil /tmp/pip-build-knaads8k/python-dateutil/.eggs/setuptools_scm-6.0.1-py3.5.egg/setuptools_scm/version.py.

[kushaldas]

I am sad to read pypa/pip#3257, not implemented yet :(

[kushaldas]

We have to install the correct version of setuptools_scm in the virtualenv created by the dh-virtualenv tool. This is managed by:

https://github.com/spotify/dh-virtualenv/blob/0.11/dh_virtualenv/deployment.py#L145-L146

Only problem is that pip does not support hash verification when used via command line.

[zenmonkeykstop]

Builds locally for me - looks like v5.0.2 has to be specified in both places above.

[conorsch]

Very nice!

looks like v5.0.2 has to be specified in both places above.

Is that so? Looks like my commit is maybe unnecessary, I'll try removing it and using only preinstall commit, and see if it still builds.

[zenmonkeykstop]

Sorry no, my point was that both are required. It would be cool to have the dependency only specified in dh_virtualenv preinstall but it needs both.

[conorsch]

Great, thanks for clarifying. Confirmed locally that only the second commit isn't sufficient for a working build.

Only problem is that pip does not support hash verification when used via command line.

Yes, that's a bummer. Given that we're bumping up the end of the support window for Xenial, though, we may be stuck with this as an option for the final releases.

[kushaldas]

Sorry no, my point was that both are required. It would be cool to have the dependency only specified in dh_virtualenv preinstall but it needs both.

Correct, because two different virtualenvs.

[kushaldas]

Yes, that's a bummer. Given that we're bumping up the end of the support window for Xenial, though, we may be stuck with this as an option for the final releases.

Only good thing is that we are not going to support Xenial long :)

[zenmonkeykstop]

Note that hash checking is not performed on this dependency, but that was already the case, as it was being installed as a build dependency via python-dateutil's setup.py.

[EdwardPrentice]

I am sad to read pypa/pip#3257, not implemented yet :(

Hi. I just stumbled across this thread and thought I might be able to help. I have ended up with this:

pip3 install -q --require-hashes -r <(echo 'mnemonic==0.19 --hash=sha256:4e37eb02b2cbd56a0079cabe58a6da93e60e3e4d6e757a586d9f23d96abea931')

Perhaps that's useful to you. An alternative could be:

pip3 install -r <(cat <<REQUIREMENTS
ansible==2.4.3.0
mnemonic==0.19
REQUIREMENTS)

All the best!

EDIT: Went one step further to give you a drive-by PR: #5903