Unpin numpy as a host dependency

[MilesCranmer]

Fixes #83

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

Wait with merging this. I found another big issue here.

[h-vetinari]

Wait with merging this. I found another big issue here.

Please remove the two lines mentioned in #86

[MilesCranmer]

#87 does this

[h-vetinari]

@conda-forge-admin, please rerender

[MilesCranmer]

Are these warnings anything to worry about?

ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-64::numpy-1.24.3-py310ha4c1d20_0, local/linux-64::pysr-0.14.1-py310hfdc917e_1
  path: 'lib/python3.10/site-packages/numpy/core/tests/test_dlpack.py'

Apart from that, things seem to be working. Are you happy with this being merged once tests pass?

[h-vetinari]

Are these warnings anything to worry about?

Yeah, this is not a good thing and needs to be fixed. It looks like your installation process is reinstalling numpy, including some of the test sources, and this gets picked up by conda's before/after-snapshotting of the host environment. The answer should be simple: don't reinstall numpy, use the one that's installed in host already.

[MilesCranmer]

It's likely this: 8fac64f (just pushed)

But, if not, there is a small chance it is from the PyCall.jl build process, which tries to install numpy if not already available: https://github.com/JuliaPy/PyCall.jl/blob/bcaba00d1e2c412b2f61d33343ef5a9ab1b9258a/deps/build.jl#L79.

[h-vetinari]

Numpy is still being spuriously installed.

But, if not, there is a small chance it is from the PyCall.jl build process, which tries to install numpy if not already available:

I haven't been involved with the julia effort in conda-forge, but this must not happen. If it cannot be solved through a build argument, then we have to patch something here. @mkitti - any experience with this issue?

[MilesCranmer]

It's interesting that this warning doesn't occur on main: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=717729&view=logs&jobId=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&j=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&t=841356e0-85bb-57d8-dbbc-852e683d1642
despite seemingly nothing else in the build process changing.

[MilesCranmer]

Are you sure that this warning message is problematic? It seems to only show that warning in the testing stage: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=717759&view=logs&j=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&t=841356e0-85bb-57d8-dbbc-852e683d1642&l=798

Some of these tests involve re-building, and building in temporary environments:

pysr-feedstock/recipe/meta.yaml

Lines 47 to 50 in 6b5be68

	- julia --project=@pysr-{{ version }} -e "using PyCall, SymbolicRegression, ClusterManagers"
	- python -c 'from pysr.sr import init_julia; init_julia()'
	- python -c 'import pysr; pysr.install(); from pysr.sr import init_julia; init_julia()'
	- python -c 'import tempfile; tempenv = tempfile.mkdtemp(); import pysr; pysr.install(tempenv); from pysr.sr import init_julia; init_julia(tempenv)'

[mkitti]

Umm... well numpy is a dependency of pysr, right? The julia-feedstock activate script should tell PyCall.jl and Conda.jl to look at the current conda environment. It should then realize that numpy is already installed, and not install a new one.

What is happening here?

[MilesCranmer]

It could be that this line is checking for numpy in the root environment, rather than the environment we are building/testing PySR in?

i.e., here is the add:

        Conda.add("numpy")

and here is a note about the default environment:

The Conda.jl package supports environments by allowing you to pass an optional
env parameter to functions for package installation, update, and so on. If
this parameter is not specified, then the default "root" environment
(corresponding to the path in Conda.ROOTENV) is used. The environment name can
be specified as a Symbol, or the full path of the environment
(if you want to use an environment in a nonstandard directory) can
be passed as a string.

For example:

using Conda
Conda.add("libnetcdf", :my_env)
Conda.add("libnetcdf", "/path/to/directory")
Conda.add("libnetcdf", "/path/to/directory"; channel="anaconda")

But I still don't understand how this should affect things, since these warnings are only displayed at the test stage?

[h-vetinari]

It's interesting that this warning doesn't occur on main despite seemingly nothing else in the build process changing.

There's a likely explanation: Before, you were using the newest numpy version (1.24.3) to build, which very likely matched whatever julia was pulling in. Now the build is using 1.21 to build, and so the file delta in $PREFIX¹ between numpy 1.21 (the "before" in the snapshot) and numpy 1.24 (the "after" as installed by julia) is what conda now thinks are files installed by pysr.

Some of these tests involve re-building, and building in temporary environments:

The ClobberWarnings happen when the environment is created, which is before these test commands are ever run.

Footnotes

1. more precisely, in $PREFIX/lib/python3<x>/site-packages/numpy, at least as far as the numpy installation is concerned ↩

[MilesCranmer]

Is there any "band-aid" fix I can apply to the build while I look into this out in the PyCall.jl side? (I don't suppose it would work as-is, would it?) Just something to fix it for users temporarily so they aren't stuck with this libstdc++ bug themselves.

Maybe I can temporarily pin libstdcxx-ng to the earlier version, finish off this PR, and then unpin it?

[h-vetinari]

You have two choices, both user-unfriendly in some way:

• keep building against numpy 1.24.3 (status quo before this PR) - which means all users of pysr need numpy>=1.24.3 at runtime, but the warnings are gone
• build against 1.21 as in this PR currently (meaning users can use any numpy >=1.21 at runtime), and introduce pretty noisy (but in this particular case relatively harmless) ClobberWarnings.

Whatever choice you make to stop the bleeding now (which is fair enough), fixing this properly should be a high priority. Randomly re-installing numpy is baaaaaaaaaaaaaaaaaaaaaaad. Please don't do it.

[h-vetinari]

If you want to go with option one, you should not revert the changes here, but additionally add the following:

Suggested change

	- numpy
	- numpy
	# temporarily rebuilding with newest numpy because PyCall.jl ends up
	# reinstalling it and we want to avoid spamming ClobberWarnings
	# TODO: remove this as soon as numpy does not get spuriously installed here anymore
	- numpy >=1.24.3

[MilesCranmer]

Was the first numpy key a typo? Or did you mean to add it two times when you said "additionally add"?

[MilesCranmer]

(I think it was a typo; let me undo this...)

[h-vetinari]

No, it was not a typo, that's exactly why I said "additionally".

One line (the unpinned one) is for being up correctly by our infrastructure (conda smithy etc.) as I explained above. The other line is a temporary, additional constraint.

[h-vetinari]

I even emphasised additionally...

[MilesCranmer]

My bad. I saw the jobs failing due to requirement errors: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=718646&view=logs&jobId=4f922444-fdfe-5dcf-b824-02f86439ef14&j=4f922444-fdfe-5dcf-b824-02f86439ef14&t=b2a8456a-fb11-5506-ca32-5ccd32538dc0 after committing this initial suggestion and just assumed it was a typo.

[MilesCranmer]

Thanks, that sounds good to me.

Whatever choice you make to stop the bleeding now (which is fair enough), fixing this properly should be a high priority. Randomly re-installing numpy is baaaaaaaaaaaaaaaaaaaaaaad. Please don't do it.

I agree! That this was even going on was news to me as well. Quite a deeply buried bug in the PyJulia infra. And I guess it makes sense we never saw any signal of this since we were building against the same numpy.

[MilesCranmer]

@conda-forge-admin, please rerender