Skip to content

feat(devpkg): configurable Nix binary cache via DEVBOX_NIX_BINARY_CACHE#2891

Open
abueide wants to merge 2 commits into
jetify-com:mainfrom
abueide:configurable-nix-binary-cache
Open

feat(devpkg): configurable Nix binary cache via DEVBOX_NIX_BINARY_CACHE#2891
abueide wants to merge 2 commits into
jetify-com:mainfrom
abueide:configurable-nix-binary-cache

Conversation

@abueide

@abueide abueide commented Jun 23, 2026

Copy link
Copy Markdown

Summary

Fixes #2599.

Devbox hardcodes https://cache.nixos.org as the Nix binary cache it queries for prebuilt package outputs. This makes devbox run fail in network-restricted environments (such as enterprise CI runners with no public internet egress), even when an internal mirror that serves the identical store paths is configured as a Nix substituter.

The hardcoded URL (internal/devpkg/narinfo_cache.go) is used in two places:

  1. The narinfo precheckreadCaches seeds its cache list with this URL, and FillNarInfoCache runs unconditionally at the start of every devbox run (via flakePlan). fetchNarInfoStatusFromHTTP does a 5s HEAD to <cache>/<hash>.narinfo; on a network error it returns the error, and fetchNarInfoStatusOnce propagates it.
  2. The fromStore of the builtins.fetchClosure expressions in the generated flake.nix.

So on a locked-down runner, devbox run fails hard with:

Error: error running script "..." in Devbox: Head "https://cache.nixos.org/<hash>.narinfo": context deadline exceeded

…even when the toolchain is fully available from an internal substituter. Setting substituters in nix.conf does not help: builtins.fetchClosure contacts its fromStore directly and ignores substituters, and the Go-level precheck has its own hardcoded URL.

This PR adds a DEVBOX_NIX_BINARY_CACHE environment variable that overrides the default cache, following the same pattern as the existing DEVBOX_SEARCH_HOST override. When unset, behavior is unchanged (https://cache.nixos.org), so this is fully backwards compatible. Pointed at an internal mirror that proxies cache.nixos.org, devbox run then works with no public egress:

export DEVBOX_NIX_BINARY_CACHE="https://nix-cache.internal.example.com/mirror"
devbox run ...

Changes

  • envir: add a DevboxNixBinaryCache (DEVBOX_NIX_BINARY_CACHE) constant to the central env-var registry.
  • devpkg: replace the binaryCache constant with a BinaryCache() accessor that resolves the env var via envir.GetValueOrDefault, defaulting to https://cache.nixos.org; readCaches uses it.
  • shellgen: the generated flake's fromStore now uses the http(s) cache the output was actually found in, via a new fetchClosureStore template helper that falls back to the configured binary cache for s3:// caches fetchClosure cannot use directly.

How was it tested?

go test ./internal/devpkg/... ./internal/shellgen/...
go tool golangci-lint run --timeout 5m
  • New unit tests: TestBinaryCache (internal/devpkg) covers the default and the override; TestFetchClosureStore (internal/shellgen) covers http(s) pass-through, the s3 fallback, and the configured fallback.
  • Existing golden-file flake tests are unchanged when the variable is unset.
  • Verified end-to-end: with DEVBOX_NIX_BINARY_CACHE set, the generated flake.nix emits fromStore = "<mirror>" and the narinfo precheck targets the mirror; unset, it remains https://cache.nixos.org.

Community Contribution License

All community contributions in this pull request are licensed to the project
maintainers under the terms of the
Apache 2 License.

By creating this pull request, I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 License as stated in
the
Community Contribution License.

Devbox hardcodes https://cache.nixos.org as the binary cache it queries for
prebuilt package outputs. This URL is used in two places:

  1. The narinfo HEAD precheck in devpkg.readCaches / fetchNarInfoStatusFromHTTP,
     which runs unconditionally on every `devbox run` (via FillNarInfoCache).
  2. The fromStore of the builtins.fetchClosure expressions in the generated
     flake.nix.

In a network-restricted environment (e.g. an enterprise CI runner with no
public egress) cache.nixos.org is unreachable, so the precheck HEAD request
times out. Because fetchNarInfoStatusFromHTTP returns the error and
fetchNarInfoStatusOnce propagates it, `devbox run` fails hard with
`Head "https://cache.nixos.org/<hash>.narinfo": context deadline exceeded`
even when an internal mirror that serves the identical store paths is
available and configured as a Nix substituter. (Setting `substituters` in
nix.conf does not help: builtins.fetchClosure contacts fromStore directly and
ignores substituters.)

Add a DEVBOX_NIX_BINARY_CACHE environment variable that overrides the default
cache, following the same pattern as the existing DEVBOX_SEARCH_HOST override
(envir.GetValueOrDefault). When unset, behavior is unchanged
(https://cache.nixos.org), so this is fully backwards compatible.

The override flows to both consumers: readCaches seeds its list with the
resolved cache, and the flake template now sets fromStore to the http(s) cache
the output was actually found in (via a new fetchClosureStore template helper),
falling back to the configured binary cache for s3 caches that fetchClosure
cannot use directly.

Pointed at an internal mirror that proxies cache.nixos.org, `devbox run` then
works with no public egress.
Copilot AI review requested due to automatic review settings June 23, 2026 23:32
@abueide abueide changed the title devpkg, shellgen: make the Nix binary cache configurable (DEVBOX_NIX_BINARY_CACHE) Make the Nix binary cache configurable via the DEVBOX_NIX_BINARY_CACHE env var Jun 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes Devbox’s Nix binary cache configurable via DEVBOX_NIX_BINARY_CACHE, replacing the previously hardcoded https://cache.nixos.org in both the Go-level narinfo precheck and the generated flake.nix builtins.fetchClosure fromStore value. This enables devbox run to work in network-restricted environments when an internal mirror/proxy is available.

Changes:

  • Introduces envir.DevboxNixBinaryCache and devpkg.BinaryCache() (defaulting to https://cache.nixos.org, overridable via env var).
  • Updates narinfo cache discovery to seed read caches from devpkg.BinaryCache() instead of a constant.
  • Updates flake generation to set fromStore based on the cache URI actually used, with tests added for both BinaryCache() and fetchClosureStore() behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
internal/shellgen/tmpl/flake.nix.tmpl Uses fetchClosureStore to dynamically set builtins.fetchClosure.fromStore.
internal/shellgen/generate.go Adds fetchClosureStore template helper and wires it into template functions.
internal/shellgen/generate_internal_test.go Adds unit tests for fetchClosureStore.
internal/envir/env.go Registers the new DEVBOX_NIX_BINARY_CACHE env var constant with documentation.
internal/devpkg/narinfo_cache.go Adds BinaryCache() and uses it to seed cache URIs instead of a hardcoded constant.
internal/devpkg/narinfo_cache_test.go Adds unit tests for BinaryCache() default/override behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/shellgen/generate_internal_test.go
Comment thread internal/shellgen/tmpl/flake.nix.tmpl Outdated
@abueide abueide changed the title Make the Nix binary cache configurable via the DEVBOX_NIX_BINARY_CACHE env var feat(devpkg): configurable Nix binary cache via DEVBOX_NIX_BINARY_CACHE Jun 23, 2026
Address review feedback:
- Render the fetchClosure fromStore through a nixString helper that escapes
  backslash, double quote, and the ${ antiquotation, so a configured cache URL
  (DEVBOX_NIX_BINARY_CACHE) with special characters can't break flake
  evaluation or be interpreted as Nix interpolation. Output is unchanged for
  ordinary URLs.
- TestFetchClosureStore now always sets DEVBOX_NIX_BINARY_CACHE (empty when
  unset) so it can't be affected by the caller's environment.
- Add TestNixString.
abueide added a commit to segmentio/analytics-react-native that referenced this pull request Jul 21, 2026
All supply-chain vectors are required to go through Artifactory's
scanning/curation proxy, not just npm - this repo's devbox/Nix
packages (nodejs, yarn-berry, jq, cocoapods, etc.) were still being
fetched straight from cache.nixos.org, unfiltered.

Adopts the same mechanism already shipped in segmentio/ajs-renderer
(#515): upstream devbox hardcodes cache.nixos.org as its fetchClosure
source and ignores any override, so a patched devbox binary is built
from source (same patch ajs-renderer uses - devbox's own
DEVBOX_NIX_BINARY_CACHE override, proposed upstream at
jetify-com/devbox#2891, not yet merged) and installed over whatever
devbox-install-action just put on PATH, immediately after every
existing "Install devbox" step in ci.yml and release.yml.

ajs-renderer's version of this involves a whole custom CI base Docker
image, an apt-installed nix-bin, and a disabled flake registry -
that complexity exists specifically to work around Twilio Buildkite's
locked-down egress. GitHub Actions runners have open internet, so
none of that bootstrap is needed here - just the patch + nix.conf
substituter + DEVBOX_NIX_BINARY_CACHE.

Verified locally before pushing:
- the patch applies cleanly to devbox 0.16.0, the version this repo's
  own devbox.lock nixpkgs-unstable pin (afce96367b) already provides
- the patched expression actually builds (nix build --impure --expr),
  and the resulting binary's compiled string table contains
  DEVBOX_NIX_BINARY_CACHE, confirming the patch compiled in rather
  than silently no-op'ing

NOT verified (needs a real CI run, flagging explicitly rather than
claiming more confidence than earned):
- whether segmentio's GitHub-Actions-OIDC-authenticated identity (or
  anonymous access) can actually reach
  https://otk.artifactory.twilio.com/artifactory/remote-nix-cache-nixos
  - this is a different Artifactory repo than the npm proxy, onboarded
  separately, and I found no IPD doc confirming general segmentio
  access to it the way trusted_builds covers npm
- whether that substituter actually serves the specific package/version
  closures devbox.lock resolves (nodejs 22, yarn-berry, cocoapods, jq,
  treefmt, nixfmt, shfmt) - if any are missing, that job falls back to
  a local build instead of a hard failure, but will be much slower
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Airgap setup with nix substituters not working

2 participants