-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Credentials provider support for builtins.fetch* #8635
Comments
This is a major blocker for adopting nix at my organization. I've made a recent push to use nix for our internal projects, but the only thing standing in the way is that we are hosting some binary inputs on a private Artifactory instance. There are multiple authentication methods but they all require setting http headers for fetchurl. I have sketched a solution that extends the tarball fetcher with support for access-tokens, but this feels like a hack and is far from ideal for the reasons described above. Since flakes depend on libfetchers, I feel this is also an extremely important feature for stabilizing flakes. |
This is also blocking our internal adoption as we cannot write derivations that can download github release assets, fetchurl actually just downloads the SSO page and thinks that is the requested target. I am not sure how far --access-tokens and --extra-access-tokens go down the chain... one idea I had was to implement a custom fetcher using the gh cli, but somehow I need to pass credentials down to the gh cli... it would be great if the access-tokens where available in the one or more of the phases... and as we use flakes arg and argstr are not an option. |
In general I don't think exposing credentials to builders is the best approach because it's easy for builders to steal your credentials, or for users to steal credentials from (remote) builders. And you can already expose credentials to builders using I think what we'd ideally want is for builtin fetchers to support authentication. If the host was configured to require authentication, then fetching would be performed on the client side and the output passed to the daemon. If the daemon was remote, then this would involve a copy. This is exactly how it works with The work @yes91 has done would be a big improvement already. Though ideally we'd want to support arbitrary headers (e.g. GitLab package registry expects a For example something similar to this: In
In
Nix would also want to allow interactivity if this required a password prompt, 2fa or something like that. |
Our use case is literally |
My understanding is if you qualify your flake with |
That is correct, however, for repo access only, not for downloading release assets as they can only be fetched using fetchurl. |
Something like access token helpers could help here. We currently use a system service to update the short-lived tokens while the developer has an active login session. Deeper credentials integration into nix seems like a good way to leak credentials into the nix store which is obviously a bad idea. Built-in fetchers ( Authentication with binary caches could also use helpers in addition to the netrc file it currently supports. There are many considerations here:
In some scenarios it's also easier to just, fetch, store and cache nix paths out-of-band. ie:
|
I would expect To get more adoption of Nix we need to keep the barrier for entry as low as possible, which means just installing nix and using a few I guess this file best describes what I am trying to achieve, but in a private setting. |
I agree. Making Nix easier to use is definitely a priority. I think this is best achieved with educational content, for now. When browsing these discussions I get a big git ano 2009 vibe. Many people wanted to get in, but also struggled to map their expectations onto git at that time. Hah, remember CVS and SVN, quite a struggle that was... To be absolutely clear this is not a you prioblem @ck3mp3r. It's very much an us issue. Just like git, nix is, probably, way to academic and abstract at it's core. I'm sure you know that. And I'm sure that some years down the road these things will be mostly non-issues. Nix is a fundamentals first kinda tool, this makes it harder to use, for now. Flakes help, but don't get us all the way. If you want to chat about how we, in our org, solve these issue for now, feel free to get in touch. |
Yes, though any helper process should be spawned by the client talking to the daemon, otherwise it wont have access to the user's session (pass, system keychain, etc). This is already an issue with S3 substituters where you have to copy the AWS credentials to the |
I have now tried --netrc-file, impureEnvVars, netrcPhase et al and am totally at a loss. Haven't found a single example that actually is reproducible and that works. Very disappointing to say the least. |
They're all pretty bad yeah. Fyi |
@szlend that is useful info, thank you, might be worth writing a devshell flake with the right cli-tools that helps with the setup. |
This is what I've come up with in case someone needs a way to access private github releases... I'd love a more elegant builtin solution, but knowing Nix's limitations I think this is as good as it gets:
I've chosen to stick to the netrc format because it also doubles as the netrc file for normal operations. |
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
Nix can now read authentication data from ~/.local/share/nix/auth/*. The files in that directory have the format: protocol=https host=my-cache.example.org username=alice password=blabla Also, the Nix daemon can now call back to the client to get authentication data from the client (if it's trusted). This makes it possible to have the daemon substitute from an authenticated binary cache, with the authentication coming from the client. Issue NixOS#8635.
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
Would the nix builder in sandbox be able to access |
I think it's not safe to set A more sophisticated approach is to save the secret to a temporary file and mount the temporary file via Suppose you have a flake to download a file from a secret Gist URL: {
description = "A flake to download a file from a URL specified in /run/secret/url.txt";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
flake-parts.url = "github:hercules-ci/flake-parts";
};
outputs = inputs@{ self, nixpkgs, flake-parts, ... }:
flake-parts.lib.mkFlake { inherit inputs; } {
systems = [ "x86_64-linux" ];
perSystem = { system, pkgs, lib, ... }: {
packages.default = pkgs.runCommand "download-file" {
buildInputs = [ pkgs.curl pkgs.cacert ];
outputHash = "sha256-6pZt7mxo+aLMFTfooiNnKkbp6oUlFEixZUQM1G/TLYA=";
} ''
url=$(cat /run/secret/url.txt)
curl -o $out $url
'';
};
};
} Then the secret Gist URL can be specified in a temporary file mounted to # A secret Gist URL
SECRET='https://gist.githubusercontent.com/Atry/9cdedb8ad653f0441f4053281bdf3d2e/raw/79bcdd7213c31076288aa795f5d4b205d1a6f4d1/flake.lock'
SECRET_FILE_NAME="$(hexdump -n 16 -e '"%02x"' /dev/urandom)".txt
SECRET_DIR="$(mktemp -d -p /tmp)"
# Only a process that knows SECRET_FILE_NAME can access the file.
chmod o+x "$SECRET_DIR"
printf %s "$SECRET" > "$SECRET_DIR"/"$SECRET_FILE_NAME"
nix build --option extra-sandbox-paths /run/secret/url.txt="$SECRET_DIR"/"$SECRET_FILE_NAME" |
I've been banging my head a while at trying to get fetchurlBoot to pick up my netrc-file set using netrc-file in my user's nix.conf as described here. I finally found that it does work if I build to a local store (--store /home/user/store), but not if the build is passed on to the daemon. Did anyone have a similar problem with the solutions posted above? Creating a global .netrc with all users' credentials would be "slightly" on the insecure side. |
Is your feature request related to a problem? Please describe.
We desperately need some sort of credentials provider support for
builtins.fetch*
functions. The best you can currently do is to usepkgs.fetchurl
, expose your credentials in plain text at some globally accessible path like/etc/nix/my-creds
and add it toextra-sandbox-paths
. You can restrict access to onlynixbld
though it's not like it matters because anyone who has access to the nix builder can echo the credentials and fetch them from the build log.Describe the solution you'd like
pass
or the system keychain).Describe alternatives you've considered
access-tokens
currently doesn't work, is limited to credentials being exposed in plain text, assumes the credentials don't need to be refreshed and is limited to oauth/pat for specific platforms like gitlab/github.Somewhat related
Priorities
Add 👍 to issues you find important.
The text was updated successfully, but these errors were encountered: