Allow custom shared projects for julia_project

[MilesCranmer]

This will fix #196. If you specify an @ in your julia_project string, like julia_project="@my_project", then this will be designated a shared project (just as if you had specified ]activate @my_project in the Julia REPL.

cc @mkitti

[MilesCranmer]

Not sure if there is any way to do this? I suppose I need to create a temp env instead (but that's a different question).

[mkitti]

You could pushfirst! a temp directory into DEPOT_PATH. Then the shared environment would be created in the temp directory. Then you can pop! the tempdir off DEPOT_PATH and recursively delete it.

[MilesCranmer]

If the DEPOT_PATH changes, and you activate a new environment, will it correctly update? It seems like it is not switching to the new setting for some reason (see error)

Traceback (most recent call last):
  File "/Users/mcranmer/Documents/PySR/test/test_env.py", line 24, in test_custom_shared_env
    self.assertEqual(cur_project_dir, potential_shared_project_dirs)
AssertionError: '/Users/mcranmer/.julia/environments' != '/var/folders/1h/xyppkvx52cl6w3_h8bw_gdqh0000gr/T/tmp8dknp59s/environments'

i.e., the current project is still in /Users/mcranmer/.julia/environments rather than the new depot

[mkitti]

Merge conda-forge/pysr-feedstock#51 to unpin openlibm then the test here will pass

[MilesCranmer]

Ah, I think I see an issue (at least, for testing this). Right now the init_julia only sets the environment via an environment variable - rather than the activate() command. The only time activate() is used explicitly is in the install part:

PySR/pysr/julia_helpers.py

Lines 86 to 88 in b6ac303

	Main.eval(
	f'Pkg.activate("{_escape_filename(julia_project)}", shared = Bool({int(is_shared)}), {io_arg})'
	)

So I think what we should actually do is call activate() if Julia has already been initialized. (Since changing the environment variable won't change the env!)

[mkitti]

So I think what we should actually do is call activate() if Julia has already been initialized. (Since changing the environment variable won't change the env!)

Correct. The only reason I used the environment variable earlier was to select the environment where we expect PyCall to be, which may not be the default environment.

[MilesCranmer]

Okay so now the DEPOT_PATH correctly sets the project. However, for some reason, julia.install() switches the environment back to the normal DEPOT_PATH... Not sure why.

from pysr.julia_helpers import init_julia, install

Main = init_julia()
Main.eval('pushfirst!(DEPOT_PATH, "/tmp/1")')
Main = init_julia("@test2")

This correctly goes to the /tmp/1/environments. However! When you do:

install("@test2")

this sets to the normal DEPOT_PATH...

  Activating project at `~/.julia/environments/test2`

Is this something in PyJulia maybe?

[mkitti]

However! When you do:

install("@test2")

this sets to the normal DEPOT_PATH...

  Activating project at `~/.julia/environments/test2`

Is this something in PyJulia maybe?

In this case it seems you have not manipulated DEPOT_PATH, so I'm guessing that DEPOT_PATH is just the default value.

Note that the DEPOT_PATH global is specific to each Julia session. Manipulating it does not influence the corresponding environment variable JULIA_DEPOT_PATH. Thus if the environment variable JULIA_DEPOT_PATH is not set and you have not manipulated DEPOT_PATH, the depot will be at its default location.

[MilesCranmer]

Thanks - the weird thing is that those commands are from the same Python session. So the DEPOT_PATH should stick around through the install() call…

[mkitti]

Is this happening on all versions of Julia or just particular ones, perhaps less than Julia 1.7.0?

I'm thinking about the subprocess and _get_julia_env_dir stuff that we did. The Julia process run by subprocess would have no idea about changes you may have made within the current Julia process.

[MilesCranmer]

Okay I think that got it. It wasn't the subprocess - it was the JULIA_DEPOT_PATH that was missing! Good catch.

[MilesCranmer]

Cool:

  Activating project at `/var/folders/1h/xyppkvx52cl6w3_h8bw_gdqh0000gr/T/tmpmrkyi7cm/environments/pysr_test_env`

Not sure what the conda errors are though.

[MilesCranmer]

Ah, the conda errors were just because the environment.yml wasn't updated to the new Julia 1.8.0 requirement for openlibm - have since unpinned it and seems to be happy.

[mkitti]

This looks like an improvement to me. I still think we should figure out how to generalize this to non-shared environments at some point.