Set JULIA_PROJECT, use Pkg.add once

[mkitti]

• Sets JULIA_PROJECT before loading pyjulia so that PyCall.jl can be contained within the pysr environment
• Also use Pkg.add in a single step to add both SymbolicRegression.jl and ClusterManagers.jl to the environment at the same time

I likely advised against using the environment variable JULIA_PROJECT in the past. However, I think this may be necessary to avoid interference from other projects if installed within the same environment.

[MilesCranmer]

Thanks! Just to check, this would still allow a user to set a custom julia_project in the PySRRegressor, right?

[MilesCranmer]

Just tested it, and it seems to work. Cool!

[MilesCranmer]

Btw, do you know how one could restart PyJulia? Right now (even before this PR), you can't change the environment after the first call to .fit, or the number of threads. Maybe there is an easy way to restart PyJulia to change these things, and load using the new environment variables.

[mkitti]

Yes. Sorry, I got distracted. Thanks for merging.

Overall, this should provide greater flexibility in terms of where PyCall.jl is installed.

[MilesCranmer]

Thanks. Not sure you saw this message, but do know if there is a way to restart PyJulia?

Also - I just updated PyPI to 0.10.2 with this merge - so feel free to update the conda-forge PR as it will now have your JULIA_PROJECT change.

[mkitti]

Btw, do you know how one could restart PyJulia? Right now (even before this PR), you can't change the environment after the first call to .fit, or the number of threads. Maybe there is an easy way to restart PyJulia to change these things, and load using the new environment variables.

Not really, although you can kind of simulate it using addprocs, which I think you kind of figured out. I'm not sure if I understand about not being able to change the environment. Certainly you can change Julia project environments via Pkg.activate.

Perhaps you mean changing environment variables such as JULIA_NUM_THREADS to increase the number of the threads?

I recall Jeff Bezanson talking about interactive threads in the State of Julia 2022 talk:
https://youtu.be/N4h46_TCmGc?t=962 (see minute 16:00)

[mkitti]

If it is not too much trouble, could you also create a Github "release" for 0.10.1 and 0.10.2?

One future change I am contemplating is whether there should be a PYSR_PROJECT environment variable that could act as a default if julia_project is None. I need to think more about how that might work with distributed computing.

[MilesCranmer]

If it is not too much trouble, could you also create a Github "release" for 0.10.1 and 0.10.2?

Done!

One future change I am contemplating is whether there should be a PYSR_PROJECT environment variable that could act as a default if julia_project is None. I need to think more about how that might work with distributed computing.

I think that is a good idea. For distributed computing I don't think it should be an issue, since the processes are all launched from within Julia, and then activated using the current project.

[MilesCranmer]

Do you know what these errors are caused by? https://github.com/MilesCranmer/PySR/actions/runs/3006737521 It looks like they are new - not sure what from. Specifically they are only on macOS/windows on Julia <= 1.6

[mkitti]

Those are likely related to my changes. Essentially PyCall has somehow failed to be installed in the current environment or we failed to change environments.

[mkitti]

Could you point me to the code for the tests that are failing?

[MilesCranmer]

I think it's failing at the install step of these tests:

PySR/.github/workflows/CI_large_nightly.yml

Line 42 in d3026af

python -c 'import pysr; pysr.install()'

[mkitti]

Here's my guess at the moment. As you found out before, not all of the Julia API recognizes that @ indicates a shared environment. The tests that are failing are on Julia versions that think JULIA_PROJECT=@pysr-0.10.3 indicates that the Julia environment exists in a local folder called @pysr-0.10.3 rather than ~/.julia/environments/pysr-0.10.3. However, with successive Julia versions this improved, which is why we do not see the issue on Julia 1.7 and beyond.

Some of the implementations then install PyCall in the local folder rather that the shared environment in the depot. But when they try to load PyCall they don't find it in the shared environment. This raises an ArgumentError similar the one I found yesterday in the conda-forge feedstock.

There is likely something else going on that accounts for the operating system dependence of the issue.

[MilesCranmer]

I see, thanks. So I guess we should add a method to revert to the old Pkg.activate if the julia version is under 1.7?

[MilesCranmer]

If I run

from julia.core import JuliaInfo
info = JuliaInfo.load()

before I set JULIA_HOME, would it already load Julia?

This is to check the version beforehand.

---

Edit: Yep, it seems fine to run this beforehand.