Focuses on worker fleet reliability. Three changes close the ways a worker could lose capacity without anyone noticing: the scheduler now tells a worker to kill an operation it no longer has executing (so an action whose clients all went away stops holding a slot), a worker that never acknowledges that kill is evicted instead of heartbeating forever, and an ABBA deadlock between worker cleanup and kill_all that could wedge an entire pool on a scheduler disconnect is fixed. Actions carrying platform properties the scheduler does not declare now match dynamically rather than being rejected, so they no longer queue forever. Anyone running remote execution should upgrade, particularly if workers have ever appeared busy while doing nothing.
What's Changed
⛰️ Features
- Tell workers to kill operations the scheduler no longer has executing on them by @amankrx in (#2693) - (a851147)
📚 Documentation
- (config-reference) regenerate for NativeLink by @github-actions[bot] in (#2700) - (aa4ca31)
- (config-reference) regenerate for NativeLink v1.6.5 by @github-actions[bot] in (#2697) - (712f321)
- Rewrite the open-source documentation site by @amankrx in (#2698) - (8ae3317)
- Generate the metrics reference and bring the config reference to v1.6.5 by @amankrx in (#2701) - (8f86dab)
- Add the journey components, page templates, lint scripts and the llms.txt generator by @amankrx in (#2699) - (b6bf4d0)
🧪 Testing & CI
- Never notify the action-done channel while holding the running-actions mutex by @amankrx in (#2706) - (4eb2d07)
- Fix a crash from scheduler/worker disconnect [1.6-patch-2] by @cormacrelf in (#2643) - (64f0fbc)
⚙️ Miscellaneous
- Evict a worker that never acknowledges a kill by @amankrx in (#2707) - (4ae820a)
- Match undeclared platform properties dynamically instead of rejecting them by @amankrx in (#2705) - (2978a1d)
- Batch input staging syscalls by @HackAttack in (#2695) - (1444869)
- Collapse the PR template comment once the description is fixed by @HackAttack in (#2703) - (5f028d9)
- Fix log dupe at cmd completion by @mridulmanohar in (#2694) - (cbcdf64)
⬆️ Bumps & Version Updates
- Update Rust crate lru to 0.18.0 [SECURITY] by @renovate[bot] in (#2681) - (5d29f82)
- Bump the pinned curl to 8.5.0-2ubuntu10.12 by @amankrx in (#2702) - (888ea82)
Upgrade notes
Two new SimpleScheduler settings ship with this release, both with working
defaults, so no configuration change is required to take the fixes:
unacknowledged_kill_timeout_s(default 60) bounds how long a worker may go
without reporting back on an operation it was told to kill before it is
evicted and its remaining work requeued.- The kill sweep itself can be tuned with
kill_revoked_operations_interval_s
(default 5) or turned off withdisable_kill_revoked_operationsfor
deployments whose long, expensive actions are better left to finish and warm
the action cache after a client disconnects.
The scheduler config rejects unknown fields, so do not add either key to a
deployment still pinned to v1.6.5 or earlier.
New Contributors
- @HackAttack made their first contribution in #2695
- @mridulmanohar made their first contribution in #2694
Full Changelog: v1.6.5...v1.6.6