A way to list environments

[layne-sadler]

The first thing I wanted to do when using Julia in VS Code was list my environments.

So I wound up on this discourse page

julia> using Pkg

julia> Pkg.envdir()
"/home/jrun/data/.julia/environments"

shell> cd "/home/jrun/data/.julia"
/mnt/data/.julia

shell> ls -la
total 36
drwxr-xr-x  8 jrun jrun 6144 Aug  9 19:52 .
drwxr-xr-x  6 jrun jrun 6144 Aug  9 18:29 ..
drwxr-xr-x  5 jrun jrun 6144 Aug  9 19:51 artifacts
drwxr-xr-x  3 jrun jrun 6144 Aug  9 19:48 compiled
drwxr-xr-x  2 jrun jrun 6144 Aug  9 19:51 logs
drwxr-xr-x 16 jrun jrun 6144 Aug  9 19:51 packages
lrwxrwxrwx  1 jrun jrun   15 Aug  9 18:29 registries -> /tmp/registries
drwxr-xr-x  3 jrun jrun 6144 Aug  9 19:52 scratchspaces
drwxr-xr-x  3 jrun jrun 6144 Aug  9 18:29 servers

Looks like there are no environments by default

Here is how I do this in python

$ pyenv versions
  system #<--- default
  3.7.6
  3.7.6/envs/GAN
  3.7.6/envs/aiqc_test
  3.8.7
  3.8.7/envs/jupyterlab3

[KristofferC]

So you want to see the equivalent of what gc --verbose outputs?

(Pkg) pkg> gc --verbose
      Active manifest files: 6 found
        `~/JuliaPkgs/PackageCompiler.jl/Manifest.toml`
        `~/.julia/environments/v1.6/Manifest.toml`
        `~/.julia/environments/v1.8/Manifest.toml`
        `~/JuliaPkgs/Pkg.jl/Manifest.toml`
        `~/.julia/environments/v1.7/Manifest.toml`
        `~/JuliaPkgs/MKL.jl/Manifest.toml`
...

Redaing what pyenv does:

Let you change the global Python version on a per-user basis.
Provide support for per-project Python versions.
Allow you to override the Python version with an environment variable.
Search commands from multiple versions of Python at a time. This may be helpful to test across Python versions with tox.

it seems to be quite different from what we call a Julia environment.

[StefanKarpinski]

I think it may be more like listing all the shared named environments? By comparison, what gc --verbose shows every environment that Pkg knows about, even local project ones. Interestingly, in Julia it turns out that people mostly use project-specific environments and not named, shared environments, whereas it seems like people use named shared environments in pyenv more? Perhaps because Julia environments are so lightweight — they're just a manifest file. By comparison, Python environments are fairly heavyweight since they are entire separate installs of Python and its libraries.

[KristofferC]

Yes, I think the shared environments are indeed pretty uncommon and that people almost always associate an environment with a piece of code that they then have next to the environment. That's why I'm a bit unsure of what should be shown here.

[fredrikekre]

#621

[mkitti]

What I would like to see is a list of shared environments in the depot stack.

For most installations this would likely just show the default shared environments for each Julia version.

Where I find this particularly useful is in situations where there is a system level configuration of one of the depots at either /usr/local/share/julia or /usr/share/julia. In this case a system administrator or a package manager may be managing one of the these depots. These depots may include standard environments shared by many users of the system or pre-installed by a package manager.

One application I have is in packaging the Python library PySR for conda-forge [1]. This is a Python wrapper for SymbolicRegression.jl. To deliver the Julia components fully installed, the build script packages a Julia depot with all the Julia dependencies as well as a shared environment. This is done by invoking pysr.install(). Upon activation of the conda environment, the depot is inserted into stack by manipulating JULIA_DEPOT_STACK.

The prepared shared environment is @pysr-0.10.3. The Python package activates this Julia environment by default. Because all the packages are available in the depot stack the user does not need to invoke pysr.install() or otherwise use Pkg.jl directly. Discovering that @pysr-0.10.3 exists somewhere on the DEPOT_PATH is difficult. Adding a command to list the shared environments in a depot stack would help make the existence of @pysr-0.10.3 easier to discover.

I can imagine that providing pre-configured shared environments could be useful for high performance computing clusters for particular common tasks or for teaching workshops.

[1] conda-forge/pysr-feedstock#43

[KristofferC]

What I would like to see is a list of shared environments in the depot stack.

You mean see the pkg> status output for all environments in the stack?

[mkitti]

I mean something like

julia> function list_shared_environments()
           println("Shared Environments")
           println("-------------------")
           for depot in DEPOT_PATH
               envdir = joinpath(depot, "environments")
               if isdir(envdir)
                   for env in readdir(envdir)
                       println("$envdir: @$env")
                   end
               end
           end
       end
list_shared_environments (generic function with 1 method)

julia> list_shared_environments()
Shared Environments
-------------------
~/.julia/environments: @__pluto_boot_v2_1.7.1
~/.julia/environments: @v1.7

I did not know that @__pluto_boot_v2_1.7.1 existed until now.

[digital-carver]

The code for Pkg.REPLMode._shared_envs is quite close to what you've written:

function _shared_envs()
    possible = String[]
    for depot in Base.DEPOT_PATH
        envdir = joinpath(depot, "environments")
        isdir(envdir) || continue
        append!(possible, readdir(envdir))
    end
    return possible
end

It seems to be explicitly marked as private with the initial underscore though - perhaps you're asking for this to be made part of the public API?

[mkitti]

The current interface to this is via autocomplete via the following keystrokes.

] activate @[tab]

That results in

(@v1.7) pkg> activate @
@__pluto_boot_v2_1.7.1 @v1.7

[mkitti]

Proposed actions:

1. Expose a Pkg.shared_envs() public API
2. Implement env command as in Introduce env command to manage environments #621
3. Implement env list subcommand using Pkg.shared_envs()

To perhaps make this more useful, perhaps there should be a way to register environments that may exist outside of the depot such that they effectively become "shared". If all operating systems had symlinks, one could just make a symbolic link into ~/.julia/environments/ but alas this is not the case.

[StefanKarpinski]

We could also list non-shared environments that we keep track of in ~/.julia/logs/manifest_usage.toml.

[zsz00]

Is there any progress so far ?

[alex180500]

bump. I felt this lack of environment routines a lot coming from python with conda

[LUK4S-B]

Would also appreciate a command to list shared environments. Now also using ] activate @[tab] as suggested above as only alternative.

[liuyxpp]

Now that I use ShareAdd.jl, I separate many useful common packages to their corresponding envs to avoid cluttering, such as @plot, @tool, @test, etc. However, sometimes I will forget whether a package has been added to one of those shared envs. Having a way to list all these share envs is really helpful.