Support artifacts-keyring integration to install pip packages from azure artifacts

[urucoder]

Problem description

Currently, Pixi does not support automated installation of packages from Azure private feeds. For our project to switch to Pixi, this feature is essential. The only workaround we've found involves hard-coding the PAT (Personal Access Token) in the index_url within the pixi.toml file, which is not a sustainable solution.

Azure feed authentication relies on keyring and artifacts-keyring. My understanding is that Pixi uses UV to manage pip installations, and UV appears to support some integration with Azure Artifacts as outlined in their documentation. However, configuring the UV_INDEX_URL environment variable does not work in Pixi.

Given that there are ongoing efforts to integrate UV with Pixi and support private feeds, I would like to request specific support for Azure Artifacts

[olivier-lacroix]

Hi @urucoder , pixi does support keyring authentication. Have a look here

the url does have to be entered in the manifest though, rather than being read from an Env Var.

[urucoder]

Hi @olivier-lacroix, thanks for the quick reply. Missed to mention that already tried with the keyring method from Pixi, but that's not enough for Azure Artifacts, they rely on artifacts-keyring for managing credentials. For example, in poetry you can do poetry self add artifacts-keyring, for Pixi there's an example on how to inject Google auth method, but nothing related to Azure. I found a discussion in the Pixi discord about this topic but there's no clear solution for a production environment

[olivier-lacroix]

You need to indeed install that azure specific package instead of the GCP specific one mentioned in the doc (keyrings.google-artifactregistry-auth)

[tdejager]

There have been people that have used this successfully with keyring in the past I think. One of the weird things is that you need VssSessionToken as username I think.

[tdejager]

Also it seems that OP of this issue had it working: #1853, would be great if we can somehow get the correct way of working documented :) Also see this for reference: https://docs.astral.sh/uv/guides/integration/alternative-indexes/#azure-artifacts

I think I would also be okay with evaluating env variables in the index-url and extra-index-url options. That would maybe also help. cc @ruben-arts (to make sure you are onboard for this)

[PanTheDev]

I have spent quite some time on this for my team and have found out quite a few details necessary to make everything work smoothly. I just came back from vacation and will make a PR documenting this shortly.

In the meantime, here are a few things that may help:

• You need to install keyring globally.
  • You need artifacts-keyring as your top priority backend for keyring.
  • The simplest way is to do pixi global install artifacts-keyring>=0.3.6
    • Prior versions have a bug causing issues with interactive auth.
  • You also need to add the keyring binary to the PATH. I do ln -s ~/.pixi/envs/artifacts-keyring/bin/keyring ~/.pixi/bin/.
• I think latest versions of Pixi can handle interactive auth while installing your envs now, but I authenticate first: keyring --mode creds get https://pkgs.dev.azure.com/<org-name>/_packaging/<feed-name>/pypi/simple/ VssSessionToken.
• You need to set the keyring-provider Pixi option to subprocess.

Hope this helps!

[jslorrma]

We are in the process of transitioning our entire project from development in Databricks to local development environments using Pixi. One of the challenges we faced was setting up the artifacts keyring. Despite following the installation and configuration guidelines, we encountered issues with the DotNet dependency of the original artifacts-keyring backend, particularly in WSL where we couldn't get it to work at all.

Inspired by this PR in the original artifacts-keyring repository, which proposed a plain Python implementation, and noting that this PR has been open since February 2023 without being merged, we decided to create our own artifacts keyring backend plugin and to open-source it. Our implementation leverages azure-identity.

Given the topic of this issue and the discussion, we wanted to share our solution here.

Please note that this package is in an early development phase. While it is already functional and has been tested, it may still contain bugs or other issues. As a result, it has not been published on the main PyPi channel yet, but is available on the test channel. For those interested, the package can be found here: keyrings.artifacts, Github

We welcome any feedback and encourage you to report any issues or bugs you encounter.

@tdejager This package is managed using pixi. If it aligns with your criteria, you might consider adding it to the list of pixi projects in your documentation.

[tdejager]

This is absolutely amazing! Thank you for this, I'll add it both to the community projects (you can make the PR is you want yourself). Will also update the documentation on Monday to link to the project.

Very cool!

[urucoder]

From my side I was able to make it work but discovered an issue while setting keyring-provider as subprocess, pixi install --pypi-keyring-provider subprocess doesn't work for me as defined in the keyring authentication.

Steps for reproducing and workaround:

pixi global install artifacts-keyring>=0.3.6
ln -s ~/.pixi/envs/artifacts-keyring/bin/keyring ~/.pixi/bin/
pixi init auth-test
cd auth-test
pixi install --pypi-keyring-provider subprocess -vvv

• This step does not create any config.toml file, and keyring does not work out of the box after this

  DEBUG pixi_config: Loading config from /etc/pixi/config.toml
  DEBUG pixi_config: Failed to load system config: /etc/pixi/config.toml (error: failed to read config from '/etc/pixi/config.toml')
  DEBUG pixi_config: Loading config from /home//.config/pixi/config.toml
  DEBUG pixi_config: Failed to load global config: /home//.config/pixi/config.toml (error: failed to read config from '/home//.config/pixi/config.toml')
  DEBUG pixi_config: Loading config from /home//.config/pixi/config.toml
  DEBUG pixi_config: Failed to load global config: /home//.config/pixi/config.toml (error: failed to read config from '/home//.config/pixi/config.toml')
  DEBUG pixi_config: Loading config from /home//.pixi/config.toml
  DEBUG pixi_config: Failed to load global config: /home//.pixi/config.toml (error: failed to read config from '/home//.pixi/config.toml')
  DEBUG pixi_config: Loading config from /home//test box/auth-test/.pixi/config.toml
  DEBUG pixi_config: Failed to load local config: /home//test box/auth-test/.pixi/config.toml (error: failed to read config from '/home//test box/auth-test/.pixi/config.toml')
  INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /home//test box/auth-test/.pixi/envs/default/conda-meta/pixi_env_prefix
  INFO pixi::lock_file::outdated: environment 'default' is out of date because it does not exist in the lock-file.
  INFO resolve_conda{group=default platform=linux-64}: pixi::lock_file::update: fetched 0 records in 921.305696ms
  INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'linux-64' in 921ms 766us 278ns
  DEBUG pixi::environment: Wrote environment file to: "/home//test box/auth-test/.pixi/envs/default/conda-meta/pixi"
  DEBUG pixi::rlimit: Attempted to set RLIMIT_NOFILE to 1024 but was already set to 1048576
  INFO pixi::environment: Creating prefix file at: /home//test box/auth-test/.pixi/envs/default/conda-meta/pixi_env_prefix
  INFO pixi::environment: Prefix file updated with: '/home//test box/auth-test/.pixi/envs/default/conda-meta'.
  INFO pixi::environment: Checking if history file exists: /home//test box/auth-test/.pixi/envs/default/conda-meta/history
  INFO pixi::environment: Creating history file: /home//test box/auth-test/.pixi/envs/default/conda-meta/history
  INFO pixi::lock_file::resolve::uv_resolution_context: using uv keyring (subprocess) provider
  ✔ The default environment has been installed.

# Create config file with keyring-provider config
echo "[pypi-config]
keyring-provider = \"subprocess\"">.pixi/config.toml

# Add private feed to project definition
# The `<user>@` section is required, it does not work if removed
echo "
[pypi-options]
index-url = \"https://VssSessionToken@pkgs.dev.azure.com/<org-name>/_packaging/<feed-name>/pypi/simple/\"">>pixi.toml

# `pixi add --pypi` does not work if python is not explicitly added to the project
pixi add python

# Then it finally works
pixi add --pypi <private-package>

[urucoder]

Also it seems that OP of this issue had it working: #1853, would be great if we can somehow get the correct way of working documented :) Also see this for reference: https://docs.astral.sh/uv/guides/integration/alternative-indexes/#azure-artifacts

I think I would also be okay with evaluating env variables in the index-url and extra-index-url options. That would maybe also help. cc @ruben-arts (to make sure you are onboard for this)

@tdejager having support for reading PIP_INDEX_URL from env variables would be great, given Azure provides PipAuthenticate@1 task for authenticate feeds in Azure Pipelines, this even remove the requirement for a keyring in Azure Pipelines