nix: flake update command exceeds GitHub API rate limit during lock maintenance

[maxbrunet]

Type of discussion.

I just want to chat

Tell us more.

The nix flake update command can fail during lock file maintenance due to the GitHub API rate limit:

Command failed: docker run --rm --name=renovate_a_sidecar --label=renovate_a_child -v "/tmp/worker/e8c44e/33f22e/repos/github/parca-dev/parca-agent":"/tmp/worker/e8c44e/33f22e/repos/github/parca-dev/parca-agent" -v "/tmp/worker/e8c44e/33f22e/cache":"/tmp/worker/e8c44e/33f22e/cache" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w "/tmp/worker/e8c44e/33f22e/repos/github/parca-dev/parca-agent" ghcr.io/containerbase/sidecar bash -l -c "install-tool nix 2.15.0 && nix     --extra-experimental-features nix-command     --extra-experimental-features flakes     flake update"
warning: '/nix/var/nix' does not exist, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store
error:
       … while updating the lock file of flake 'git+file:///tmp/worker/e8c44e/33f22e/repos/github/parca-dev/parca-agent?ref=refs/heads/main&rev=c779e52e93d1dad60d6813473c22986d3f6e80f2'

       … while updating the flake input 'flake-compat'

       … while fetching the input 'github:edolstra/flake-compat'

       error: unable to download 'https://api.github.com/repos/edolstra/flake-compat/commits/HEAD': HTTP error 403

       response body:

       {"message":"API rate limit exceeded for 34.239.12.110. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

The Nix configuration file has an option to map hosts to their auth token called access-token:

access-tokens = github.com=23ac...b289 gitlab.mycompany.com=PAT:A123Bp_Cd..EfG gitlab.com=OAuth2:1jklw3jk

https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-access-tokens

I suppose Renovate should pass down its GitHub token (and any other host tokens it has?) to Nix. The places where Nix reads its config are listed at the top of the above linked documentation page.

The current code:

renovate/lib/modules/manager/nix/artifacts.ts

Lines 25 to 29 in 60ac8c7

	if (config.isLockFileMaintenance) {
	cmd = `nix \
	--extra-experimental-features nix-command \
	--extra-experimental-features flakes \
	flake update`;

Here is a live example: parca-dev/parca-agent#1570

cc @JamieMagee

[viceice]

https://github.com/containerbase/base/blob/740d313ba7d78f6f5077ccc90bb2823e6a78a5d9/test/nix/Dockerfile#L42

we already do it on containerbase tests 🙃

[rarkins]

@nabeelsaabna can you confirm whether the tokens the app uses are prefixed or not?

[nabeelsaabna]

The tokens from APP are not prefixed
example token: ghs_sDoQBOAIumFXRsSXHamjteqeOeIIzR0UjiOy (not a valid one of course 😄)

[flurie]

If this is universally broken, can #21568 be reverted pending a fix?

[maxbrunet]

Thank you @nabeelsaabna. I have been able to reproduce the issue locally with a GitHub App Installation token, and have a fix for it: PR #23884

[dotboris]

I'm hitting this as well using the Renovate GH on GitHub.

Here's the error I'm getting: dotboris/alt#319 (comment)
Job log: https://developer.mend.io/github/dotboris/alt/-/job/7d4ef34c-419c-4085-99df-5d78747929ce
Renovate config: https://github.com/dotboris/alt/blob/cf1f2177cec684281c08edcb634524b5d734ecf0/renovate.json5

Full error

{
  "err": {
    "cmd": "/bin/sh -c docker run --rm --name=renovate_a_sidecar --label=renovate_a_child -v \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\":\"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" -v \"/tmp/worker/01bb5f/8a3b67/cache\":\"/tmp/worker/01bb5f/8a3b67/cache\" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" ghcr.io/containerbase/sidecar bash -l -c \"install-tool nix 2.16.1 && nix     --extra-experimental-features nix-command     --extra-experimental-features flakes --extra-access-tokens github.com=**redacted** flake update\"",
    "stderr": "warning: '/nix/var/nix' does not exist, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store\n\r\u001b[K\r\u001b[Kerror:\n       … while updating the lock file of flake 'git+file:///tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt?ref=refs/heads/main&rev=63cf88294fab30d41550010601025fdbdd209954'\n\n       … while updating the flake input 'flake-utils'\n\n       … while fetching the input 'github:numtide/flake-utils'\n\n       error: unable to download 'https://api.github.com/repos/numtide/flake-utils/commits/HEAD': HTTP error 401\n\n       response body:\n\n       {\n         \"message\": \"Bad credentials\",\n         \"documentation_url\": \"https://docs.github.com/rest\"\n       }\n",
    "stdout": "[23:58:51.110] INFO (9): Installing tool nix v2.16.1...\n[23:58:51.113] INFO (9): Preparing legacy tool nix ...\ninstalling v2 tool nix v2.16\nlinking tool nix v2.16\nnix (Nix) 2.16.2\nInstalled v2 //usr/local/containerbase/tools/v2/nix.sh in 2 seconds\n[23:58:53.642] INFO (9): Installed tool nix in 2532ms.\n",
    "options": {
      "cwd": "/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt",
      "encoding": "utf-8",
      "env": {
        "HOME": "/home/ubuntu",
        "PATH": "/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "LC_ALL": "C.UTF-8",
        "LANG": "C.UTF-8",
        "BUILDPACK_CACHE_DIR": "/tmp/worker/01bb5f/8a3b67/cache/containerbase",
        "CONTAINERBASE_CACHE_DIR": "/tmp/worker/01bb5f/8a3b67/cache/containerbase"
      },
      "maxBuffer": 10485760,
      "timeout": 900000
    },
    "exitCode": 1,
    "name": "ExecError",
    "message": "Command failed: docker run --rm --name=renovate_a_sidecar --label=renovate_a_child -v \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\":\"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" -v \"/tmp/worker/01bb5f/8a3b67/cache\":\"/tmp/worker/01bb5f/8a3b67/cache\" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" ghcr.io/containerbase/sidecar bash -l -c \"install-tool nix 2.16.1 && nix     --extra-experimental-features nix-command     --extra-experimental-features flakes --extra-access-tokens github.com=**redacted** flake update\"\nwarning: '/nix/var/nix' does not exist, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store\n\r\u001b[K\r\u001b[Kerror:\n       … while updating the lock file of flake 'git+file:///tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt?ref=refs/heads/main&rev=63cf88294fab30d41550010601025fdbdd209954'\n\n       … while updating the flake input 'flake-utils'\n\n       … while fetching the input 'github:numtide/flake-utils'\n\n       error: unable to download 'https://api.github.com/repos/numtide/flake-utils/commits/HEAD': HTTP error 401\n\n       response body:\n\n       {\n         \"message\": \"Bad credentials\",\n         \"documentation_url\": \"https://docs.github.com/rest\"\n       }\n",
    "stack": "ExecError: Command failed: docker run --rm --name=renovate_a_sidecar --label=renovate_a_child -v \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\":\"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" -v \"/tmp/worker/01bb5f/8a3b67/cache\":\"/tmp/worker/01bb5f/8a3b67/cache\" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w \"/tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt\" ghcr.io/containerbase/sidecar bash -l -c \"install-tool nix 2.16.1 && nix     --extra-experimental-features nix-command     --extra-experimental-features flakes --extra-access-tokens github.com=**redacted** flake update\"\nwarning: '/nix/var/nix' does not exist, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store\n\r\u001b[K\r\u001b[Kerror:\n       … while updating the lock file of flake 'git+file:///tmp/worker/01bb5f/8a3b67/repos/github/dotboris/alt?ref=refs/heads/main&rev=63cf88294fab30d41550010601025fdbdd209954'\n\n       … while updating the flake input 'flake-utils'\n\n       … while fetching the input 'github:numtide/flake-utils'\n\n       error: unable to download 'https://api.github.com/repos/numtide/flake-utils/commits/HEAD': HTTP error 401\n\n       response body:\n\n       {\n         \"message\": \"Bad credentials\",\n         \"documentation_url\": \"https://docs.github.com/rest\"\n       }\n\n    at ChildProcess.<anonymous> (/opt/containerbase/tools/renovate/35.144.2/node_modules/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:525:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)"
  }
}