Conda-Forge Julia interop

[mkitti]

The julia-feedstock at conda-forge allows one to install julia as part of the conda environment. In this case, perhaps we would should set environmental variables such that CondaPkg.jl uses the same conda environment. How do we do this?

https://github.com/conda-forge/julia-feedstock/blob/master/recipe/scripts/activate.sh

[cjdoris]

Currently there's no option to do this - CondaPkg always creates a Conda environment per Julia project. Of course this behaviour could be changed given a good reason.

With the Conda distribution of Julia, are you allowed to use multiple Julia projects per Conda environment? I'm not sure it makes sense to tie these all to the same Conda environment, since the point of projects is to have independent dependencies.

I'm also a bit wary of special-casing the behaviour of CondaPkg depending on which Julia distribution one is using.

[mkitti]

In the 1.7 branch, via the above activate.sh script, we create a new Julia depot per conda environment. What we basically need are two settings:

1. An environmental variable to indicate the location of the conda executable, via Introduce CONDA_JL_CONDA_EXE environment variable Conda.jl#216 this will CONDA_JL_CONDA_EXE. Notably this could either point to conda or mamba
2. An environmental variable to indicate the prefix of the conda environment. This is currently CONDA_JL_HOME

One option would be just to use the same variables as Conda.jl. Another might be to use the more modern Preferences.jl.

[cjdoris]

Ok, I'll think about adding an "external" backend to allow the user to use a pre-existing Conda environment, triggered by an environment variable.

Such a backend would differ quite a bit from the default behaviour of CondaPkg. First, it would use the same Conda environment for all Julia projects. Second (and less important), it would never remove unneeded packages. As such I'm not convinced you should set this as the default behaviour. That's up to you though.

[ngam]

I'm not sure it makes sense to tie these all to the same Conda environment, since the point of projects is to have independent dependencies.

I'm also a bit wary of special-casing the behaviour of CondaPkg depending on which Julia distribution one is using

I think what we are really advocating for here is just to avoid reinstalling conda/mamba. So I don't think you should change anything in your setup, except to use the conda environment already existing as base. From there, you can make as many conda/julia envs as you want (we also added stacking support).

Does that make sense? In my view, this is part is not necessary:

Such a backend would differ quite a bit from the default behaviour of CondaPkg. First, it would use the same Conda environment for all Julia projects.

How does your CondaPkg handle the base conda environment? Just have it take as the conda env where the julia from conda-forge (from julia-feedstock) is installed. So, let's say someone makes a conda env called jenv, then does conda install julia (or in one line, conda create -n jenv julia). This will isolate julia inside the jenv environment, also creating a julia environment/project called jenv instead of v1.7. I think what will make most sense is for you to take the conda env jenv as your base and then from there continue as you'd like --- all would be nested and isolated inside that $CONDA_PREFIX ad infinitum (unless the user specifies otherwise).

[ngam]

A more interesting question for me is how does your package handle shared dependencies? Say, conda install llvm and the julia-vendored LLVM. There is now an increasing number of these, and that's a major headache in the julia-feedstock these days.

[ngam]

To be slightly more specific, I think only control-flow-type additions are needed around:

here:

https://github.com/cjdoris/CondaPkg.jl/blob/01b17fb1129a945a51bccfe26ce11e0fbf65884d/src/CondaPkg.jl#L11

here:

https://github.com/cjdoris/CondaPkg.jl/blob/01b17fb1129a945a51bccfe26ce11e0fbf65884d/src/env.jl#L11-L27

here:

https://github.com/cjdoris/CondaPkg.jl/blob/01b17fb1129a945a51bccfe26ce11e0fbf65884d/src/env.jl#L74-L81

The last part tells me you're essentially setting up a new base and relying on it to branch out. This could be changed by replacing $MAMBA_ROOT_PREFIX to something like $CONDA_PREFIX and condabin to bin. But it should only do so if it detects that we are inside a conda env to begin with, hence the environment variable suggested by @mkitti.

I guess though, you'd likely have to change a lot more because you've built this entire thing around this MAMBA (I am not sure where it comes from) so maybe it is better for us to target interop upstream with mamba and then it will trickle down here. Does that sound like a better idea, @cjdoris?

[cjdoris]

I think there's some confusion in terminology, because Conda conflates two separate things:

• The base environment, which is the default Conda environment.
• The root prefix, which is where Conda caches packages so that installing the same thing into N different environments doesn't download it N times.
  In Conda (and Mamba I think), the base environment and the root prefix are the same directory. In MicroMamba there is no base environment: the root prefix is NOT typically a Conda environment.

Presently, CondaPkg.jl uses the MicroMamba executable and root prefix supplied by the MicroMamba.jl package. With this, it creates a Conda environment specific to the current Julia project.

One thing it would be easy to add would be the ability to configure CondaPkg.jl to use a different executable and root prefix (which I'll call the "backend"). I was always intending to add multiple backends to CondaPkg.jl but haven't yet.

For example, there could be a "SystemConda" backend which uses whichever conda is already in the PATH, and the root prefix it is already configured with. Being Conda, the root prefix and the base environment are the same thing, but CondaPkg.jl doesn't care that the root prefix is an environment and doesn't use it. Except that CondaPkg.jl doesn't deactivate any existing Conda environment, so the project-specific environment it creates is effectively stacked on top of whichever environment was activated before launching Julia (usually the base environment, or more likely jenv in your example above).

What I initially thought you were asking for was the ability to force CondaPkg.jl not to create its own project-specific environment, but instead use some fixed Conda environment. This pretty much goes against the central feature of CondaPkg.jl (separation of dependencies between projects) so I'm much less in favour of this.

[ngam]

Presently, CondaPkg.jl uses the MicroMamba executable and root prefix supplied by the MicroMamba.jl package. With this, it creates a Conda environment specific to the current Julia project.

Okay, I see. Thanks for explaining this --- obviously my response above was conflating micromamba with mamba 😅

Wouldn't be easier if we just talk to the MicroMamba.jl team and have them set their root prefix as jenv above in the case when in a the conda-forge's julia? We would need to trigger an installation of micromamba or something as well...

[cjdoris]

I am the MicroMamba.jl team! 😄 The job of that package is very simple - to give you a working micromamba installation isolated from the rest of your system, so it's not the place for these changes.

However it will be simple to add a "SystemConda" backend to CondaPkg.jl to use a pre-installed Conda (and its root prefix) instead of using MicroMamba.jl. It would be activated by setting the env var JULIA_CONDAPKG_BACKEND=SystemConda.

[ngam]

I am the MicroMamba.jl team! 😄

Major plot twist 🤯

[cjdoris]

OK it was simple so I did it.

You can now set JULIA_CONDAPKG_BACKEND=System to use a pre-installed conda (or mamba or micromamba) instead of the one from MicroMamba.jl.

Not released yet, but on the main branch if you want to try it out.

[cjdoris]

Closing since I added the backend. Let me know if you have more questions.

[mkitti]

Thanks! Has this been released yet?

[cjdoris]

Good point, I've just triggered a release.