sciunit fails to build with pip 20.1.x (with Python 3.9.0b1 and 3.8.3 alike).

[sanjayankur31]

Hi there,

While testing sciunit builds in Fedora 33, we see that it fails to build with the new pip version, since it uses the pip internal API:

+ /usr/bin/python3 setup.py build '--executable=/usr/bin/python3 -s'
Traceback (most recent call last):
  File "/builddir/build/BUILD/sciunit-0.2.2/setup.py", line 48, in <module>
    install_requires=read_requirements(),
  File "/builddir/build/BUILD/sciunit-0.2.2/setup.py", line 28, in read_requirements
    reqs = [str(ir.req) for ir in install_reqs]
  File "/builddir/build/BUILD/sciunit-0.2.2/setup.py", line 28, in <listcomp>
    reqs = [str(ir.req) for ir in install_reqs]
AttributeError: 'ParsedRequirement' object has no attribute 'req'

This is with sciunit-0.2.2.

Downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=1838486

The complete build logs are attached:
sciunit-build-log.txt
sciunit-root-log.txt

Sciunt is now included in NeuroFedora, and will be included in the next Comp-Neuro Fedora lab release (along with neuronunit which is a WIP):

• https://docs.fedoraproject.org/en-US/neurofedora/compneuro-tools/
• https://labs.fedoraproject.org/en/comp-neuro/

[appukuttan-shailesh]

I too had the same issue yesterday with trying to update docs on ReadTheDocs for a package that uses SciUnit. As a quick fix, I reverted to a previous version of pip (20.0.2). But we would need to eventually resolve this going forward.

More here: pypa/pip#8188

[rgerkin]

@sanjayankur31 @appukuttan-shailesh Pip changes their API with every version for no reason, dropping key functionality. We have a fix in another fork which I will merge in here shortly.

[rgerkin]

@sanjayankur31 @appukuttan-shailesh
Merged b5f8d99 into master here, tested on latest pip, and released on PyPI as version 0.2.2.2.

[sanjayankur31]

Thank you. I'll update the Fedora package now.

The Fedora Python-SIG that filed the bug had noted that:

sciunit's setup.py uses internal pip API: from pip._internal.req import parse_requirements

So, I checked pip out of curiosity, and it seems that they do not have a stable external API:
https://github.com/pypa/pip/blob/master/NEWS.rst#1000b1-2018-03-31

The docs say:

https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

As noted previously, pip is a command line program. While it is implemented in Python, and so is available from your Python code via import pip, you must not use pip’s internal APIs in this way.

I don't know what a replacement for this would be, though. I see stuff like pip-api is available:
https://pypi.org/project/pip-api/

[rgerkin]

@sanjayankur31 Using the pip API solved some practical problems and edge cases (like handling comments in requirements.txt files, parsing dependencies from git repos with specific version numbers) and a bunch of other projects were doing it, but I won't get burned again! I didn't realize they weren't even trying for stability. If we keep requirements.txt clean it shouldn't be needed.