Enable specifying the location of the environment outside the project directory

[mjkanji]

It would be great if project-specific envs could be installed outside of the project directory.

The use case here is that I currently use a cloud provider that creates ephemeral machines with free (block) storage, while the project files (i.e., code) are stored on a persistent EFS-like server that is mounted onto the ephemeral VMs. This is a typical managed-JupyterLab setup.

As such, if the environment was saved under the root of my project, I'd be paying quite a bit for storage. My currently set up uses micromamba, where the environment is saved in the root directory in a Docker container (which uses the machine's storage), and the project files (code) are mounted on top.

Having a way to tell pixi where to save the environment or to link an (already existing) external environment with a Pixi project would be a welcome addition.

[baszalmstra]

This is a good idea. The environment can always be recreated from a project/lock files, so any pixi command can work with both an existing as well as a missing environment. This makes the pixi essentially stateless.

For anyone looking to implement this, we should:

• add a --target-dir CLI argument (similar to --manifest-path) that indicates where the environment files should be placed. If the argument is not specified ./.pixi is used. All commands that interact with an environment should have this argument.
• Additionally we could decide to allow configuring this argument through an environment variable (PIXI_TARGET_DIR) as well to make it easy to capture a default in docker image layer.

Im not hung up on the names of these arguments/environment variables.

[mjkanji]

How about also adding an option in pixi.toml itself so that one doesn't have to keep passing an additional CLI arg/env variable every time. If the env variable is defined in the bash profile, then all pixi projects read the same value, which would definitely be problematic.

Configuring the --target-dir via the TOML file let's you customise per repo without making CLI usage more verbose.

[baszalmstra]

But wouldn't the --target-dir be very machine specific?

[mjkanji]

But wouldn't the --target-dir be very machine specific?

Sure, but there are certainly ways to make it invariant by using either relative paths (relative to the project repo or pixi.toml) or constants (like ($HOME|~)/some/path). At least, that's what I had in mind with the suggestion.

[baszalmstra]

I think it's important for projects to be isolated as much as possible. I wouldn't want to clone a random project only to notice that the installation of my environment now happens in a completely "random" location on my PC. I would even consider that a security risk.

Sure running a task has the same implications but that's one step further from installing the dependencies.

If we can guarantee that the target-dir in the project file is relative to the root that would be fine.

[wolfv]

I think we can have some "private" configuration variables in teh .pixi folder (just like git does?).

[huzwares]

Hey, I'd love to take a shot at resolving this. Mind if I hop in and give it a try? Assigning it to me would be awesome. Thanks! 🦀

[baszalmstra]

Sure thing! Let me know if you need any help!

[huzwares]

Thank you for assigning me the task. I'll begin working on it tomorrow. If I encounter any challenges, I'll be sure to reach out for assistance.

[melund]

I have different use case which unfortunatly also needs this feature.

My manager wants to place some python scripts in folder on OneDrive. The scripts are supposed to run on some excel file located there.

I can't create a pixi.toml file, because any environment .pixi/envs folder would be synced.

If the pixi.toml file could specify that the environment should be placed Temp that would be nice.

[pavelzw]

If the pixi.toml file could specify that the environment should be placed Temp that would be nice.

You can specify this in .pixi/config.toml with https://pixi.sh/dev/reference/pixi_configuration/#detached-environments

[melund]

Fantastic. Thanks a ton @pavelzw. I looked around but didn't realize that this was already implemented.

[ruben-arts]

Close as fixed by #1381