feat: Persistant shell manifests #1080

abkfenris · 2024-03-28T14:40:54Z

Using Project::discover to detect if Pixi's manifest path has been set by an environment variable in another pixi shell instance. PIXI_IN_SHELL is also set to be explicit that this should be an activated environment.

A warning is emitted before a command when the project is loaded if the discovered path is different from the one given by the environment variable, along with a warning after most commands in case their outputs are longer and scroll the initial warning off the terminal.

This is an alternate take on #1055

fixes: prefix-dev#1038 Using `Project::discover` to detect if Pixi's manifest path has been set by an environment variable in another `pixi shell` instance. `PIXI_IN_SHELL` is also set to be explicit that this should be an activated environment. A warning is emitted before a command when the project is loaded if the discovered path is different from the one given by the environment variable, along with a warning after most commands in case their outputs are longer and scroll the initial warning off the terminal. This is an alternate take on prefix-dev#1055

abkfenris · 2024-03-28T14:46:47Z

I haven't added docs or tried to get environments to load (#893) yet, but I can if this is more what the Prefix team is looking for in comparison to #1055

baszalmstra

I think this makes sense!

We would also need to do this for the environment name. If it is not set we should read it from the env (if PIXI_IN_SHELL is 1).

baszalmstra · 2024-03-28T14:57:15Z

src/project/mod.rs

+        if std::env::var("PIXI_IN_SHELL").is_ok() {
+            if let Ok(env_manifest_path) = std::env::var("PIXI_PROJECT_MANIFEST") {
+                if env_manifest_path.as_str() != project_toml.to_str().unwrap() {
+                    tracing::warn!(
+                        "Using manifest {} from `PIXI_PROJECT_MANIFEST` rather than local {}",
+                        env_manifest_path,
+                        project_toml.to_string_lossy()
+                    );
+                    return Self::load(Path::new(env_manifest_path.as_str()));
+                }
+            }
+        }


I would change this to only display a warning if the discovered manifest is not equal to the manifest from the environment.

That's what this should do. Are you finding an instance when it's warning more than it should?

Warning when local doesn't match the manifest from environment variable

pixi on  manifest-in-shell is 📦 v0.17.1 via 🦀 v1.76.0 took 5s ❯ pixi shell --manifest-path /Users/akerney/GMRI/NERACOOS/NERACOOS_ERDDAP_K8S/pixi.toml . "/var/folders/n6/hl07fcc55ys7mbybt01gb8r80000gn/T/pixi_env_LkI.sh" pixi on  manifest-in-shell is 📦 v0.17.1 via 🦀 v1.76.0 ❯ . "/var/folders/n6/hl07fcc55ys7mbybt01gb8r80000gn/T/pixi_env_LkI.sh" (sea-eagle) pixi on  manifest-in-shell is 📦 v0.17.1 via 🦀 v1.76.0 via 🅒 sea-eagle ❯ pixi info WARN pixi::project: Using manifest /Users/akerney/GMRI/NERACOOS/NERACOOS_ERDDAP_K8S/pixi.toml from `PIXI_PROJECT_MANIFEST` rather than local /Users/akerney/Geek/pixi/pixi.toml Pixi version: 0.17.1 Platform: osx-arm64 Virtual packages: __unix=0=0 : __osx=14.4.1=0 : __archspec=1=arm64 Cache dir: /Users/akerney/Library/Caches/rattler/cache Auth storage: /Users/akerney/.rattler/credentials.json Project ------------ Version: 0.1.0 Manifest file: /Users/akerney/GMRI/NERACOOS/NERACOOS_ERDDAP_K8S/pixi.toml Config locations: Last updated: 28-03-2024 10:18:19 Environments ------------ Environment: default Features: default Channels: conda-forge Dependency count: 28 Dependencies: python, cftime, dagster-postgres, dagster-k8s, dagster, dask, fsspec, ioos_qc, pandas, pydantic, pystac, python-frontmatter, yaml, requests, sentry-sdk, shapely, xarray, netcdf4, pint, ipykernel, matplotlib, erddapy, uvicorn, xpublish-edr, fastapi, xpublish, xpublish-opendap, uxarray Target platforms: osx-arm64 Tasks: install-kernel WARN pixi::project: Used manifest /Users/akerney/GMRI/NERACOOS/NERACOOS_ERDDAP_K8S/pixi.toml from `PIXI_PROJECT_MANIFEST` rather than local /Users/akerney/Geek/pixi/pixi.toml (sea-eagle)

Quiet when they are the same.

pixi on  manifest-in-shell [+] is 📦 v0.17.1 via 🦀 v1.76.0 took 5s ❯ pixi shell . "/var/folders/n6/hl07fcc55ys7mbybt01gb8r80000gn/T/pixi_env_2hk.sh" pixi on  manifest-in-shell is 📦 v0.17.1 via 🦀 v1.76.0 ❯ . "/var/folders/n6/hl07fcc55ys7mbybt01gb8r80000gn/T/pixi_env_2hk.sh" (pixi) pixi on  manifest-in-shell is 📦 v0.17.1 via 🦀 v1.76.0 via 🅒 pixi ❯ pixi info Pixi version: 0.17.1 Platform: osx-arm64 Virtual packages: __unix=0=0 : __osx=14.4.1=0 : __archspec=1=arm64 Cache dir: /Users/akerney/Library/Caches/rattler/cache Auth storage: /Users/akerney/.rattler/credentials.json Project ------------ Manifest file: /Users/akerney/Geek/pixi/pixi.toml Config locations: Last updated: 28-03-2024 09:26:37 Environments ------------ Environment: default Features: default Channels: conda-forge Dependency count: 7 Dependencies: pre-commit, rust, openssl, pkg-config, git, cffconvert, tbump Target platforms: linux-64, win-64, osx-64, osx-arm64 Tasks: install, build, lint, test, test-all, bump Environment: docs Features: docs, default Channels: conda-forge Dependency count: 13 Dependencies: mkdocs, mkdocs-material, mdx_truly_sane_lists, pillow, cairosvg, mike, pre-commit, rust, openssl, pkg-config, git, cffconvert, tbump Target platforms: osx-arm64, win-64, osx-64, linux-64 Tasks: install, docs, deploy-dev, deploy-latest, test-all, build, lint, bump, test, build-docs Environment: schema Features: schema, default Channels: conda-forge Dependency count: 12 Dependencies: python, pytest, jsonschema, pydantic, pyyaml, pre-commit, rust, openssl, pkg-config, git, cffconvert, tbump Target platforms: osx-64, win-64, linux-64, osx-arm64 Tasks: bump, test-all, build, generate-schema, install, test, lint, test-schema

src/project/mod.rs

ruben-arts

I don't have time to do a complete review but I found one small thing.

I already like this better, probably best to close the other PR.

src/cli/shell.rs

abkfenris · 2024-03-28T16:30:50Z

Ok, I've taken a look at how to get the environment name from the environment variables. I'm thinking I can add a method to EnvironmentName that will replace the section in a bunch of commands that loads or maps it:

pixi/src/cli/list.rs

Lines 102 to 104 in 36ac560

    
           let environment_name = args 
        
               .environment 
        
               .map_or_else(|| EnvironmentName::Default, EnvironmentName::Named);

    /// Tries to read the environment name from an argument, then it will try
    /// to read from an environment variable, otherwise it will fall back to default
    pub fn from_arg_or_env_var(arg_name: Option<String>) -> Self {
        if let Some(arg_name) = arg_name {
            return EnvironmentName::Named(arg_name);
        } else if std::env::var("PIXI_IN_SHELL").is_ok() {
            if let Ok(env_var_name) = std::env::var("PIXI_ENVIRONMENT_NAME") {
                return EnvironmentName::Named(env_var_name);
            }
        }
        EnvironmentName::Default
    }

Which would allow those lines in list.rs to become

let environment_name = EnvironmentName::from_arg_or_env_var(args.environment);

I'm a little confused about what's different with the pixi run command and how it loads an environment.

pixi/src/cli/run.rs

Lines 64 to 73 in 36ac560

    
           let explicit_environment = args 
        
               .environment 
        
               .map(|n| EnvironmentName::from_str(n.as_str())) 
        
               .transpose()? 
        
               .map(|n| { 
        
                   project 
        
                       .environment(&n) 
        
                       .ok_or_else(|| miette::miette!("unknown environment '{n}'")) 
        
               }) 
        
               .transpose()?;

fixes: prefix-dev#893 Currently doesn't work for `pixi run <task>`.

src/project/mod.rs

…is no local manifest

ruben-arts

It works now 👍

ruben-arts · 2024-04-05T14:43:35Z

Thanks @abkfenris !!

abkfenris mentioned this pull request Mar 28, 2024

feat: Persistant shell manifests (using CLI env) #1055

Closed

baszalmstra reviewed Mar 28, 2024

View reviewed changes

abkfenris added 2 commits March 28, 2024 11:00

Lets Windows borrow the shell environment

7ec06e9

Merge remote-tracking branch 'upstream/main' into manifest-in-shell

eac80f2

ruben-arts requested changes Mar 28, 2024

View reviewed changes

src/cli/shell.rs Outdated Show resolved Hide resolved

abkfenris and others added 6 commits March 28, 2024 12:41

Move shell env var to activation

f32d67c

Load environment from env vars when in shell

7e983fa

fixes: prefix-dev#893 Currently doesn't work for `pixi run <task>`.

Merge branch 'main' into manifest-in-shell

2dbf2b8

Merge branch 'main' into manifest-in-shell

81e4b71

Update tree to use EnvironmentName::from_arg_or_env_var

3af938e

Warn after tree as well

d12c7e8

abkfenris marked this pull request as ready for review April 4, 2024 00:51

ruben-arts reviewed Apr 4, 2024

View reviewed changes

src/project/mod.rs Show resolved Hide resolved

abkfenris added 2 commits April 5, 2024 07:38

Merge remote-tracking branch 'upstream/main' into manifest-in-shell

deb3c1f

Make sure that discover can use a manifest from the shell when there …

e692605

…is no local manifest

ruben-arts approved these changes Apr 5, 2024

View reviewed changes

baszalmstra approved these changes Apr 5, 2024

View reviewed changes

ruben-arts merged commit 526486c into prefix-dev:main Apr 5, 2024
23 of 24 checks passed

renan-r-santos mentioned this pull request Apr 6, 2024

feat: add support for persistent Pixi shells renan-r-santos/pixi-kernel#5

Merged

ruben-arts mentioned this pull request Apr 8, 2024

feat: Default to PIXI_ENVIRONMENT_NAME in pixi list #900

Closed

pavelzw mentioned this pull request Apr 8, 2024

pixi list should default to the shell-environment if active #893

Closed

ruben-arts mentioned this pull request Apr 10, 2024

Support setting the --environment flag via env var #889

Closed

BrewTestBot mentioned this pull request Apr 10, 2024

pixi 0.19.0 Homebrew/homebrew-core#168524

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Persistant shell manifests #1080

feat: Persistant shell manifests #1080

abkfenris commented Mar 28, 2024

abkfenris commented Mar 28, 2024

baszalmstra left a comment

baszalmstra Mar 28, 2024

abkfenris Mar 28, 2024

ruben-arts left a comment •

edited

abkfenris commented Mar 28, 2024

ruben-arts left a comment

ruben-arts commented Apr 5, 2024

feat: Persistant shell manifests #1080

feat: Persistant shell manifests #1080

Conversation

abkfenris commented Mar 28, 2024

abkfenris commented Mar 28, 2024

baszalmstra left a comment

Choose a reason for hiding this comment

baszalmstra Mar 28, 2024

Choose a reason for hiding this comment

abkfenris Mar 28, 2024

Choose a reason for hiding this comment

ruben-arts left a comment • edited

Choose a reason for hiding this comment

abkfenris commented Mar 28, 2024

ruben-arts left a comment

Choose a reason for hiding this comment

ruben-arts commented Apr 5, 2024

ruben-arts left a comment •

edited