skrog 0.8.0

Admission control learns to ask where an image came from, the engine start path
finally says where its time goes, and a chain of silent failures gets found —
several of them by running the product rather than by reading the diff, and
four of them by an independent review of this release's own code.

Added

• skrog doctor catches a prune.every that stops the engine ever idling
  (#496). The automatic prune
  talks to the engine through Skrog's own pipe, and that traffic restarts the
  idle window — so a prune interval at or below idle-timeout resets the clock
  before it can expire and the RAM is never returned.

  [warn] idle-stop versus automatic prune: prune.every 10m is not longer than
         idle-timeout 30m, so the engine will never idle-stop
         fix: set prune.every longer than idle-timeout:
              `skrog config set prune.every 2h` (or lower idle-timeout)
  

  Both settings are off by default, so nothing ships in this state — it takes
  turning both on and setting the prune shorter, which is the opposite of how
  either is normally tuned. But the symptom is "my engine never releases its
  RAM", which people notice, cannot explain, and have no reason to connect to a
  housekeeping setting.

  Judged from the configuration, not from the idle-stop counters. The
  counters look like the obvious evidence and are the weaker signal: a machine
  that has simply been busy also shows zero idle stops. Two durations and a
  comparison is a verdict; a counter at zero is a guess.

  The underlying fix — having the prune reach the engine without crossing the
  pipe — is still open. The issue's cheap route (wsl -d skrog-engine docker …)
  turns out not to exist: there is no docker client in the engine distro,
  only the daemon. The workable version talks the engine API directly, the way
  the idle probe and provenance already do, and wants its own release window
  rather than this one.

• skrog stop --supervisor (#482).
  There was no supported way to stop the supervisor, and the installer told
  people to use one: refusing to overwrite a running skrog.exe, it said
  "Stop it first: skrog stop". skrog stop is the engine only — by design
  and by its own usage text — so the supervisor and its watchdog kept the
  binary locked, the user followed the instruction, retried, and got the
  identical error with no next step. Every in-place upgrade hit this.

  The flag stops the engine, then the supervisor. The watchdog goes with it
  without being asked, because a clean exit is final (internal/watchdog) —
  the same property that stops it racing restart --supervisor. The installer
  now names the command that works.

  Still the rarer intent by far, which is why it is a flag and not the
  default: nearly everything anyone wants from stop is the engine, and a
  supervisor that survives is what makes skrog start quick and keeps the pipe
  where it was.

• skrog doctor checks the emulation you asked for is actually live
  (#480).
  skrog config set emulation.platforms linux/arm64 reports success and takes
  effect on the next engine start — where registration can fail. When it did,
  the only evidence was a WARN in supervisor.log: skrog restart printed
  "engine is running", and the container then died with exec format error,
  which is the exact symptom the setting was turned on to remove.

  [warn] foreign-architecture containers: emulation.platforms asks for
         linux/arm64, but no handler is registered for arm64
  

  Separate from the cross-architecture builds check next door, which reads
  the same table. That one asks what can this machine do and never warns,
  because a standing warning about a capability nobody uses is what teaches
  people to skim. This one asks did what you asked for happen, is silent
  unless you asked, and warns when the answer is no. The first cannot catch
  this: with no handlers at all it reports "the default builder is native-only
  … nothing is wrong here", which on a machine that requested emulation is
  wrong twice.

  The remedy names the likeliest cause, which is not guessable: the emulator
  ships in the engine image, and images before 29.8.1-3 do not carry it
  (#479).

• deny-unattributable-images: refuse an image this machine has no record of
  (#343).
  allow-registries judges a reference, and a reference is a label anyone
  can reattach. docker load a tarball, docker tag it
  registry.example.com/app:1, and every reference-based rule passes on bytes
  that never came from registry.example.com.

  Skrog now records where each image came from — keyed by image ID, written
  after a pull it allowed — and with this rule set, refuses to run one it has
  no record of. Keyed by ID is the whole point: retagging does not create
  provenance.

  Opt-in, and inert without allow-registries, exactly like
  deny-unattributable-builds; skrog policy show says so rather than letting
  you believe otherwise. It also reports the recorded/pre-existing split,
  because turning this on without knowing how much of a machine's image set has
  a record is how a security feature gets switched off again an hour later.

  Images you already had are trusted. The first start with this present
  records the existing image set as pre-existing, once. Refusing a machine's
  entire cache on upgrade is not a defensible default, and the count is
  reported rather than implied.

  It is drift protection, not a boundary, and the docs say so. Anyone who
  can run wsl -d skrog-engine -u root docker load talks to the engine
  directly and nothing at the pipe sees it (#418). What it catches is the
  ordinary accidental laundering: a tarball copied off a laptop, a docker save from a machine with different rules, an image left behind by a setup
  nobody remembers. It also fails open — an engine it cannot reach or a
  record it cannot read allows the request, because a container that will not
  start because a lookup timed out is the worse outcome.

  It refuses your own builds too. The build is not judged, but the image it
  produces has no recorded origin, so the docker run after it is refused — at
  the pipe a BuildKit build is an opaque gRPC stream with nothing to attribute
  it to. So this is a rule for machines that consume images (a CI runner, a
  locked-down workstation), not ones that build them. docs/policy.md says so
  where someone deciding whether to turn it on will read it, rather than
  leaving them to find out from a refusal.

  Applied by every listener that serves the engine, not just the supervisor's
  pipe: skrog serve and skrog proxy too, for the reason
  #257 gives.

• skrog logs where an engine start spent its time
  (#398). One line per start:

  engine start phases probe=17ms prelaunch=1.085s launch=17ms
                      dockerdReady=2.96s agent=0s shareSocket=129ms total=4.206s
  

  #404 had to reconstruct this from the timestamps of three unrelated log
  lines, and a 1.2 s stretch sat unexamined in the gap between two of them for
  two releases. Anyone asking "where does a start go" now gets the answer from
  one run instead of arithmetic on a log.

  What it immediately showed is that there is no five-second win in Skrog's
  own code. Of a ~4.2 s start: ~1.1 s is the WSL distro booting, ~2.6–3.3 s is
  dockerd starting itself, and Skrog's own share — the health probe, the launch
  call, the socket share — is about 160 ms. Two optimisations were written
  against this issue and measured; one is in this release and one was reverted,
  both described below.

  It also settled the ROADMAP item that has carried "measure the real number"
  since v0.2: the idle-wake path is 4.8–6.3 s, against 5.0–8.4 s for
  a full stop/start. The wake is not meaningfully cheaper, and the reason
  the README gave for expecting it to be was wrong — an idle stop is
  wsl --terminate, the same operation skrog stop performs, so the wake pays
  the same distro boot and the same dockerd startup. There is no cheaper resume
  path to reach for. The README and ROADMAP now say the measured numbers.

• The engine agent starts alongside the wait for dockerd, not before it
  (#398). Provisioning the agent
  secret and launching the agent are ~430 ms of wsl round trips, and they sat
  between launching dockerd and starting to wait for it — while dockerd was
  already busy taking ~3 s to come up. They now overlap that wait, and the
  phase log reads agent=0s on every start.

  Still fully started before StartEngine returns, on every exit path
  including the failure one: nothing may observe a half-started agent, because
  the symptom would not be a crash but a silent fall back to the socat relay at
  ~165 ms per connection instead of ~0.6 ms.

  End to end this is worth about 100 ms, not 430 — median 4.50 s → 4.20 s
  over ten paired runs — because dockerd's own startup is the binding
  constraint and the agent work was already hiding inside it. Reported that way
  rather than quoting the 430 ms the phase log removes.

Changed

• skrog version and skrog doctor name the rootfs revision, not just its
  checksum (#484).

  rootfs   29.8.1-3  (f1be49d99b9a...)
  

  It used to be the truncated SHA-256 alone — a value that identifies the bytes
  exactly and answers none of the questions asked of it. Which revision am I
  on, and is it the one that carries the emulator, were unanswerable from the
  output of the two commands whose job is to answer them. doctor was worse:
  it printed version 29.8.1 one line above, so the two lines together gave
  the engine version twice and the revision never.

  The checksum stays, because it is what --rootfs-sha256, the manifest and a
  bug report speak, and it is the only identifier an image installed from
  outside the manifest has. Without a ref the line degrades to the checksum
  alone rather than inventing one.

  version --json gains engineRef as a new field; rootfsSha256 and
  version are untouched, because docs/cli-json.md pins them and a reader
  switching on either must keep working.

  The naming rule moved to internal/release on the way. It had one caller
  when it was written and now has four, and a rule with four copies is a rule
  that drifts.

Security

• The registry cache's upstream is judged by policy, and may not be plain
  HTTP (#421).
  skrog cache enable --upstream <url> wires a mirror into the engine, and
  dockerd applies registry-mirrors to every unpinned Docker Hub pull
  without changing the reference. So a mirror aimed at any host served that
  host's content under an allowed name, and allow-registries passed because
  the reference was never the thing in question.

  The upstream is now checked against the effective allowlist, and http:// is
  refused unless --insecure is passed — a plaintext mirror for Docker Hub is
  a content-substitution position on every unpinned pull on the machine, held
  by anyone on the network path.

  Three places in the code and docs asserted this could not happen, on the
  ground that "a mirror changes where bytes come from, not which image was
  asked for". The premise is true and the conclusion does not follow: where the
  bytes come from is what a registry allowlist is for. All three are corrected
  rather than deleted.

• A machine policy file that a standard user could have written is refused
  (#418). C:\ProgramData ships
  with BUILTIN\Users:(CI)(WD,AD) and CREATOR OWNER:(OI)(CI)(IO)(F), so on
  any machine where a fleet policy has not landed yet a standard user can
  create ProgramData\skrog first and own it. Loading was a bare os.ReadFile
  with no owner check, so the "machine layer" could be the user's own rules
  wearing its authority — worse than no machine layer, because skrog policy show reported it as in force.

  The owner must be Administrators or SYSTEM. Owner and not the full DACL,
  deliberately: the hole is an ownership one, and a hand-written ACE walker
  that gets an edge case wrong fails a correctly deployed fleet silently.

  skrog policy show reports provenance now, which is the honest half of
  the promise. The layer cannot be made unbypassable — the supervisor runs as
  the user — so what it can do is not lie about which bypass happened:

  machine rules: C:\ProgramData\skrog\policy.yaml  REFUSED
    ignored because it is owned by CONTOSO\alice, not by Administrators or SYSTEM.
  

  and the documented SKROG_MACHINE_POLICY_DIR redirect, which used to be
  indistinguishable from a machine that simply had no fleet policy:

  machine rules: none found
    SKROG_MACHINE_POLICY_DIR redirects the machine layer to C:\Users\me\empty,
    which holds no policy.yaml.
  

  policy show --json gains machineProvenance for the fleet-dashboard case:
  a machine whose policy was refused is the one worth an alert, and it looked
  identical to one that never had any.

  The third route in #418 — wsl -d <distro> -u root, proxy --no-path-translation, wsl-integrate — is unchanged and still documented
  as a limit. It is not fixable in this package.

Fixed

• Findings from an independent pre-release review. Two reviewers were run
  over everything since v0.7.1 with no knowledge of why any of it was written.
  Between them they found four defects serious enough to have held the release,
  all of them in code added this cycle:

  • The reconciler held its lock across the adopted-engine repair, with no
    timeout — reintroducing #437
    on a path added days earlier. A wslservice that stopped answering would
    have parked the supervisor while holding mu, so every docker command
    hung instead of failing and skrog status could not even report it. The
    repair now runs outside the lock and bounded, like the health probe beside
    it, and a test holds it in flight and checks that Demand still answers.

  • The provenance engine round trip had no timeout at all. resolveTimeout
    bounded only the dial; after that it blocked in ReadResponse with no
    deadline, and the vsock dialer clears the deadline it used for its own
    handshake before handing the connection over. An engine that accepted the
    connection and wedged would hang docker run forever with no output, and —
    on the recording path, which runs on the relay goroutine — leave the
    bridge's client count above zero so idle-stop was vetoed for the life of
    the process. It goes through an http.Client now, the way the container
    probe already did.

  • An unreadable store refused every container, which is the exact
    opposite of what this page and the changelog both promised. read returned
    nil for any failure and Lookup could not tell "no record" from "could
    not read"; sc.Err() was never checked either, so one over-long line
    silently discarded every record after it. Read failures are now an error
    all the way up and the request is allowed, with skrog policy show saying
    the store could not be read rather than quietly reporting zero.

  • Pre-existing images were never seeded on the ordinary first boot.
    Seeding was gated on the store existing, and the store is also created by
    the first recorded pull — so on the normal sequence (supervisor starts
    before the engine, pull creates the store) the seed short-circuited
    forever and a machine's whole pre-upgrade image cache stayed
    unattributable, silently. It has its own marker file now.

  Also from the review: compaction could evict the record of an image that is
  still installed (a base image pulled months ago, refused today), so the
  180-day expiry is gone and compaction asks what the engine still holds before
  evicting anything; ReapplyRunning always returned nil, making the
  supervisor's warning about a failed re-apply dead code; and four tests
  asserted less than their names claimed — including the one added for the
  previous release's Close fix, which never reaches the line it was written
  for on Windows.

• logSafe is a sanitizer CodeQL can see again. Rewriting it to bound its
  output instead of its input (previous entry) moved the untrusted rune into a
  variable computed before the unicode.IsControl guard and overwritten
  inside it — identical at runtime, invisible to taint analysis. CodeQL stopped
  recognising the function as a barrier and re-raised go/log-injection on all
  three call sites, which is how it was found.

  The rune is written only inside the guarded branch now. A sanitizer a checker
  cannot see is one that stops being checked the next time someone edits it.

• skrog --help lines up. The command list was padded to a fixed width of
  10 and healthcheck is 11, so that row had no gap at all and wsl-integrate
  overflowed by three. The column is derived from the longest name now, with a
  three-space gap, and a test fails if any name collides.

  The settings list in skrog config --help had the same problem differently:
  its column was baked into a format string per row, at 22 for some keys and 26
  for others, and gpu and gpu.vendor were missing from it entirely —
  settable, documented nowhere, and therefore absent from docs/reference.md
  too. Both lists now render through one helper.

• A supervisor that adopts a running engine re-applies its settings
  (#501). Found on a real machine
  while writing the multi-platform docs for this release: emulation.platforms
  was set, skrog config get echoed it back, the supervisor log said the
  handlers were registered, and docker run --platform still failed with the
  exec format error the setting exists to remove. The binfmt_misc table was
  empty.

  binfmt_misc handlers are kernel state, the socket share is a VM-level
  bind, and the agent is a separate process — any of them can be gone while
  dockerd itself is perfectly healthy. StartEngine has an already-running
  branch that re-applies exactly this, and its comment says it exists for "a
  supervisor that finds a healthy engine". Nothing reached it: the supervisor
  called Start only when the engine was down.

  So the exposed paths are the ones with no command behind them — the watchdog
  restarting after a crash, and logon autostart finding an engine still up.
  skrog start and skrog restart --supervisor were never affected, because
  those commands call StartEngine themselves, which is precisely why this was
  invisible to anyone testing by hand.

  Re-applied once, on adoption, not every tick: that branch is four wsl
  round trips and paying it at the health interval would be the worse bug. A
  table cleared later, under a supervisor that keeps running, is still only
  reported — by skrog doctor (#480) —
  and not repaired.

• A scheduled prune is part of the supervisor's lifetime
  (#423). It ran on a bare
  goroutine that nothing could wait for, cancel or recover from, with three
  consequences:

  Shutdown did not stop it. On skrog restart, a logoff or a service stop, the
  reconciler returned while a docker system prune -a kept deleting for up to
  thirty minutes, with the process about to exit underneath it. Run now waits
  for a prune in flight, bounded at five seconds — long enough for a sweep that
  is finishing to record its clock, short enough that a wedged prune cannot
  hold a logoff open. After that it is abandoned to its own timeout, which is
  the deliberate outcome rather than an error.

  A panic in it took the bridge down. Prune is caller-supplied and ran with
  no recover, so a bad implementation killed the supervisor and every docker
  command with it. A failed prune is a disk that stays full; a dead supervisor
  is a machine where docker stops working. Recovered and reported now, with the
  guard still cleared so one bad sweep cannot disable pruning until restart.

  An interrupted prune re-ran immediately. Killed before it recorded the clock,
  the old value stood and the next supervisor was instantly due — a prune on
  the first healthy tick after every logon. The clock is written before the
  sweep as well as after, which changes the failure from "prunes too often" to
  "may skip one interval", the safer direction for a feature whose rules are
  about not deleting things unexpectedly.

• skrog engine upgrade no longer races the supervisor
  (#486). It failed
  intermittently, with exit status 1 and no message, on a different file each
  time — and succeeded every time with the supervisor stopped. 0.7.1 shipped it
  as a known issue with "run it again" as the remedy.

  The upgrade already wrote desired=stopped before the swap, with a comment
  saying it was to keep the supervisor out of the way. Honoring that state is
  what caused the failure. The reconciler, seeing "stopped but up", calls
  Terminate — and terminating a WSL distro kills every process in it,
  including the wsl --exec copying binaries into that same distro. The exec
  itself boots the distro, dockerd comes up with it, and the next tick sees
  exactly the state that makes it terminate. Both sides agreed on the goal and
  fought over the route.

  "Stop the engine" and "keep your hands off the distro" are different
  instructions, and the second could not be written as a desired state. So
  there is now a maintenance hold: while it is held the reconciler does
  nothing — not even probe, since the probe boots the distro it asks about. It
  carries an expiry, because a crashed holder must not leave a supervisor that
  has silently stopped reconciling forever; that would present as an engine
  that never recovers with nothing in any log to say why.

  Four upgrades in a row on the machine that reproduced it, supervisor running
  throughout, all clean.

• A restart after engine upgrade, compact, relocate or a snapshot
  restore no longer drops emulation, GPU, proxy and host CA settings
  (#490). Found while validating
  the fix above: after an upgrade, docker run --platform linux/arm64 failed
  with exec format error on a machine where it had just worked.

  The supervisor re-reads every per-start setting and has since #83 and #202.
  Every one-shot command that restarts the engine built its own options and
  read no config at all, so the engine came back with emulation off, no GPU
  spec, no proxy and no imported CAs — until something else restarted it.

  Emulation is the one that fails loudly. On a corporate network the others are
  worse: pulls stop working, or fail on a certificate, some time after an
  unrelated maintenance command, with nothing connecting the two.

  There is one list now, used by the supervisor and by all seven one-shot
  paths. The bug was that there were two — the supervisor's, which was right,
  and everyone else's, which did not exist.

• skrog doctor no longer reports WSL's own COM stub as an injected
  third-party module (#488).

  [warn] injected modules: 1 third-party module(s) are loaded into this process
         wslserviceproxystub.dll  (C:\Program Files\WSL\wslserviceproxystub.dll)
       fix: ... ask whoever manages the agent for an exclusion
  

  There is no agent and nobody to ask. That DLL is Microsoft-signed, ships with
  WSL, and Skrog loads it itself: it talks to wslservice over COM (#380),
  and Windows maps the marshalling stub into any process that does. Doctor said
  as much two lines above, in the WSL service access check.

  The filter treated only C:\Windows\... as the platform's own, and modern
  WSL is serviced separately and installs to %ProgramFiles%\WSL. It now
  exempts a directory named wsl, matched as a whole path segment so that
  something shipping in wslhook\ is still reported — the point is to quieten
  one component, not to hand anything WSL-shaped a free pass.

  This fired on every healthy machine, which is the specific way a check whose
  value is that its output means something becomes a check people skim.