Proposal: fully offline hk via an embedded pkl package (seed the persistent cache)

[sahidvelji]

Context

This follows up on discussion #1198 ("How do I run hk entirely offline"), where two working options were laid out:

1. Vendor the pkl package into the repo — download hk@X.zip once while online, extract to .hk/pkl/hk-X/, rewrite the two imports in hk.pkl to local paths, and commit. Strongest today: a cold machine needs no network. Costs: per-repo setup, ~650 KB committed into every repo, imports diverge from the canonical package:// form, and it must be re-run on every version bump.
2. Persistent package cache — since shipped as feat(pkl): persist packages for offline evaluation #1199 (v1.56.0): HK_PKL_CACHE_DIR + HK_PKL_OFFLINE. Once a package is on disk, evaluation survives config-cache invalidation and works offline. Costs: still needs one online cold-warm, and the cache must survive between runs.

Summary

Both options above leave the same gap: the cold start. A fresh environment that has never warmed the cache (option 2) or vendored the package (option 1) still has to fetch https://github.com/jdx/hk/releases/download/vX/hk@X.zip (which 302-redirects to release-assets.githubusercontent.com) on the very first evaluation — exactly what fails or is unavailable in restricted/corporate networks and on cold CI runners.

This proposes a third option: embed the matching-version pkl package in the hk binary and seed the persistent cache from it before evaluation. Unlike vendoring, nothing is committed per-repo and imports stay in canonical form; unlike the persistent cache alone, no online cold-warm is needed. It is an extension of #1199, not a competing mechanism — the embedded copy simply pre-populates the cache #1199 already manages, for the common case where hk.pkl pins the running binary's version.

Why the cache-seed approach fits pklr as-is

Reading pklr 1.4.0 (src/eval.rs), the persistent cache is a flat, content-addressed store — much simpler than Apple pkl's ~/.pkl/cache layout:

• resolve_package_uri maps hk's config imports (Format 3) to a single zip URL:
  • package://github.com/jdx/hk/releases/download/v1.56.0/hk@1.56.0#/Config.pkl
    → PackageSource::Zip("https://github.com/jdx/hk/releases/download/v1.56.0/hk@1.56.0.zip", "Config.pkl")
  • Both amends …#/Config.pkl and import …#/Builtins.pkl resolve to the same zip — the exact release asset hk already builds via pkl project package pkl and uploads.
• package_cache_paths(cache_dir, url, ext) stores two files under <cache_dir>/packages/:
  • <fnv1a64(url):016x>.zip — the raw zip bytes
  • <fnv1a64(url):016x>.url — the original URL (collision guard)
  • No metadata JSON, no sha256. validate_package_bytes for a zip only checks that it opens and every entry reads, which the real artifact passes.
• fetch_package_bytes reads the cache first and, when offline, short-circuits before any network call. So a seeded cache + HK_PKL_OFFLINE=true means zero network, deterministically.

So "embed the package" reduces to "write the embedded zip into the cache in pklr's format, keyed by the URL hk's own version resolves to."

Proposed change: two small, coordinated PRs

PR 1 — pklr: expose a preload_package method

Reuse the two functions that already exist (resolve_package_uri, write_package_cache) so the cache layout and hash scheme stay private to pklr and callers never depend on them:

/// Preload a package into the persistent cache from in-memory bytes so that a
/// later resolution of `uri` needs no network. No-op if no package_cache_dir
/// is configured.
pub fn preload_package(&self, uri: &str, bytes: &[u8]) -> Result<()> {
    let (url, ext) = match resolve_package_uri(uri)? {
        PackageSource::Zip(url, _)        => (url, "zip"),
        PackageSource::Direct { url, .. } => (url, "pkl"),
    };
    validate_package_bytes(&url, ext, bytes)?;
    self.write_package_cache(&url, ext, bytes);
    Ok(())
}

(Exact signature/placement — Evaluator method vs. also a builder passthrough — is yours to decide; this is the shape.)

PR 2 — hk: embed the package and seed the cache

Before the evaluator runs (src/config.rs, run_pklr), seed the cache for the binary's own version. No need to parse hk.pkl — unconditionally seeding the running binary's version is correct: if the config pins that version (the common case) it's a cache hit and never fetches; otherwise the entry is simply unused.

static EMBEDDED_PKG_ZIP: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/hk-pkl.zip"));

if *env::HK_PKL_EMBEDDED && !EMBEDDED_PKG_ZIP.is_empty() {
    let v = version::version();
    let uri = format!(
        "package://github.com/jdx/hk/releases/download/v{v}/hk@{v}#/Config.pkl"
    );
    let _ = evaluator.preload_package(&uri, EMBEDDED_PKG_ZIP); // best-effort
}

Plus a HK_PKL_EMBEDDED env toggle (default on) so it can be disabled, and docs/CHANGELOG.

The build step (the only fiddly part)

Builtins.pkl is generated and git-ignored (scripts/gen_builtins.py), and the embedded artifact is the built package zip, so the raw pkl/ tree can't be embedded directly. A small build.rs would stage hk@{version}.zip into $OUT_DIR/hk-pkl.zip:

• In release CI, copy the already-built package zip into place before cargo build.
• For local dev builds where the zip isn't staged, fall back to an empty payload so embedding no-ops (guarded by the is_empty() check above).

Scope / size

• pklr: ~15 lines + a small test. Reuses existing internals.
• hk: ~30 lines of logic, a 2-line env toggle, a ~25–40 line build.rs, plus docs. Net ~80–120 lines, dominated by build-time artifact plumbing rather than logic.

Behavior and trade-offs

• Only helps the version-match case — when hk.pkl pins the same version as the running binary. That is the common case (release tooling rewrites configs to match, and users pin to what they installed). A config pinning a different version still fetches that version normally. Embedding is a cold-start optimization, not a replacement for the package:// model.
• No behavior change when versions differ or when disabled (HK_PKL_EMBEDDED=0).
• Binary size: the package is ~650 KB and compresses well — negligible for the binary.
• Couples the schema to the binary only for the embedded copy; the canonical package://…/releases/download/vX source of truth is unchanged, and non-matching versions resolve exactly as today.

[jdx]

Thanks for the detailed proposal. I think the release-asset plumbing is manageable: the asset name, URL, and contents are predictable from the hk version, so we should be able to generate or stage the matching archive as part of the build.

My main hesitation is the exact-version coupling. The embedded package only provides a cold-start offline path when hk.pkl pins the same version as the running binary. Anyone intentionally running a newer binary against an older package pin would still need the persistent cache or vendoring. I would frame this as a matching-version offline fallback rather than fully offline support across arbitrary pins.

That said, the size and runtime cost look small, and mismatched versions can retain the current behavior. I would be open to reviewing a PR. It should preserve existing cache/custom rewrite precedence, fall back normally on version mismatch, and test both a matching cold offline start and the mismatch case.

AI-assisted — Tool: Codex; model: openai/gpt-5; version: unavailable.

[dkowis]

Could there be a syntax in the package import to say "always import the packaged version" so that it doesn't require a configuration update every time? Does that cause too many other problems?

[jdx]

Pkl is not my language, and I would not be in favor of adding special syntax to pklr for this. If version selection belongs in package import syntax, you could propose that feature to Pkl upstream. pklr could then support the upstream behavior instead of introducing a pklr-specific extension.

AI-assisted — Tool: Codex; model: openai/gpt-5; version: unavailable.

[dkowis]

That makes sense. Is there another way to avoid the specific version being in the configuration file?

[jdx]

not that I can think of