CI: manually install numpy==1.19.4 to prevent release candidate

[sphuber]

Fixes #4614

This is the usual problem that occurs when numpy releases a candidate
release. Even though we specify limits to not install this, pymatgen
installs it as a dependency in their build process where limits are
ignored and so the latest release gets installed. In this case
numpy==1.20.0rc1 is getting installed and that dropped support for
Python 3.6 and so our builds on that version brick.

The workaround is to manually install the compatible version, in this
case numpy==1.19.4 before installing aiida-core. This way pymatgen
will simply use that in their build process.

[sphuber]

Thinking about this, I will actually have to base this of release/1.5.1 because those tests will also be broken and we need to release that tomorrow or early next week.

[codecov]

Codecov Report

Merging #4615 (f057efb) into release/1.5.1 (b9fcbd8) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@                Coverage Diff                @@
##           release/1.5.1    #4615      +/-   ##
=================================================
- Coverage          79.51%   79.50%   -0.00%     
=================================================
  Files                482      482              
  Lines              35320    35320              
=================================================
- Hits               28080    28078       -2     
- Misses              7240     7242       +2     

Flag	Coverage Δ
django	73.67% <ø> (ø)
sqlalchemy	72.84% <ø> (-<0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiida/engine/daemon/client.py	72.42% <0.00%> (-1.14%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9fcbd8...f057efb. Read the comment docs.

[chrisjsewell]

Does this relate to a particular open issue on pymatgen, so I can go and voice my displeasure lol

Also, should we add a note on this somewhere in the documentation; in the install or troubleshooting section?

[sphuber]

Does this relate to a particular open issue on pymatgen, so I can go and voice my displeasure lol

No current one, but we have dealt with this problem already at least 4 times. Each time numpy makes a new minor version release essentially. We have to introduce this explicit limit again to get everything running. At the time, we opened this issue which got rejected because it concerned numpy removing Python 2.7 support and the pymatgen developers didn't want to add fixes for this. It seems that this is not just a Python 2.7 problem but a generic problem with their build requirement definitions. Not sure if they would be amenable to considering addressing this time around. We could maybe open an issue again and ask at least.

For reference the commits I could quickly find that addressed the exact same problem in the past:

f5d6cba
227d30f
a7575d8

Also, should we add a note on this somewhere in the documentation; in the install or troubleshooting section?

Would be a good idea since it is happening often. At least by know I am able to recognize the problem almost instantly and add the workaround in place without wasting a lot of precious time, but we should document this. Still other developers should then still be aware of this phenomenon. In any case, given that it concerns aiida-core developers, it shouldn't go in the docs but the wiki if anything.

[chrisjsewell]

On a related note @csadorf we can remove the 2020-resolver now:

WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0.

[chrisjsewell]

In any case, given that it concerns aiida-core developers, it shouldn't go in the docs but the wiki if anything.

Will this not then affect people running pip install aiida-core[atomic_tools] then?

I was actually just trying to reproduce this locally, but then saw that its not an issue for Macs, because there is a pymatgen wheel (materialsproject/pymatgen#1520 (comment)). Macs are clearly the best 😝

[csadorf]

In any case, given that it concerns aiida-core developers, it shouldn't go in the docs but the wiki if anything.

Will this not then affect people running pip install aiida-core[atomic_tools] then?

Yes, it does: https://github.com/aiidateam/aiida-core/runs/1496526194?check_suite_focus=true

It is quite frustrating that this is problem is arising yet again. Those are the actions I would recommend:

1. Create an issue on pymatgen outlining the problem and its community impact once more.
2. If we know a solution, create a pull request (or at least offer to create one) that resolves or mitigates the issue.
3. Add a warning to the installation documentation that warns about this issue, specifically related to "Python 3.6" and the atomic_tools extra.

For clarification, restricting our dependencies to versions of pymatgen that support the same Python versions that we support with markers does not solve the problem, correct?

[sphuber]

For clarification, restricting our dependencies to versions of pymatgen that support the same Python versions that we support with markers does not solve the problem, correct?

Indeed, this does not solve the problem, because the limitations we impose through the setup.json are not respected by setuptools which is what pymatgen runs to build itself. So when we ask to install pymatgen it will build taking the latest available version of numpy (including release candidates and pre-releases) unless some version already is installed in the environment

[csadorf]

I've made some suggestions on how to make this work-around fix a bit less intrusive, but overall it is of course fine, at least for the ci-code workflow. We can't have this impede our dev work. I'm a bit more hesitant to accept this for the test-install workflow.

Either way, I want to be pragmatic about this, so the fix is fine, even without my proposed changes, albeit I would strongly recommend to at least accept the in-line comments.

[csadorf]

Let's at least ensure that the version from the requirements files is installed and add a comment as to why this work-around is necessary:

Suggested change

	pip install numpy==1.19.4
	# Work-around issue caused by pymatgen's setup process, which will install the latest
	# numpy version (including release candidates) regardless of our actual specification:
	pip install `grep 'numpy==' requirements/requirements-py-${{ matrix.python-version }}.txt`

[sphuber]

For reference: the final solution that got merged here was not ideal. It hardcoded numpy==1.19.4 whereas it should just get the version from the requirements file for the corresponding Python version, because that is guaranteed to be compatible. The same problem needed to be addressed on support/1.4.x in order for us to be able to release v1.4.4, so we applied the same fix but in the correct general way. This fix was also applied to release/1.5.1 and replaced the commit merged in this PR. The new commit is 8058a97.

[mkhorton]

Indeed, this does not solve the problem, because the limitations we impose through the setup.json are not respected by setuptools which is what pymatgen runs to build itself. So when we ask to install pymatgen it will build taking the latest available version of numpy (including release candidates and pre-releases) unless some version already is installed in the environment

Is this not expected behavior, and the responsibility of setuptools to make that determination?

If we know a solution, create a pull request (or at least offer to create one) that resolves or mitigates the issue.

This would definitely be appreciated and in the spirit of open source. As mentioned more specifically in the #4614 issue and the referenced issue in pymatgen, we have been unable to identify or reproduce the issue.

More broadly, for creating re-producible builds, I can recommend poetry since pip alone can often be very brittle. One thing that also comes to mind is that pip 20.3 has also made significant changes in how its dependency resolver works, so maybe that could be causing issues.

[sphuber]

@mkhorton thanks a lot for following up on this, really appreciate it. I responded to the issue over at pymatgen with some additional information on how we encounter the problem and where I think the problem comes from, but it is still not entirely clear. I propose we continue the discussion there and see if we can nail down the origin of the problem and hopefully come up with a solution.