Repository navigation
Major changes
- Memory escalation now recovers from pod-level OOM kills. When a worker is lost whole (for example an out-of-memory kill that takes the pod down before any usage is reported), the scheduler now steps the action's memory reservation up on re-dispatch once the same action has lost
escalate_after_worker_lossesworkers (default 2,0disables), instead of re-dispatching at the same size until the loss budget fails it. Workers also report each running action's peak memory on the keepalive, so an action that was observed over its reservation escalates on the first loss, and a memory-kill escalation now steps past the sampled peak rather than just past the old reservation. (#3168, #3170 by @amankrx) - macOS builds of the source tree work again. The Linux-only
capsdependency introduced in #3166 is now gated to Linux, restoringcargoand Bazel builds on macOS and theaarch64-apple-darwinrelease artifact. (#3174 by @corcillo)
Upgrade and rollback
- Loss-driven memory escalation is on by default. Actions that repeatedly lose workers for non-memory reasons (spot preemption, node drains) inflate their reservations too; set
escalate_after_worker_losses: 0undermemory_escalationfor the v1.8.4 behavior. Older binaries, including v1.8.4, reject a config that names the new field — keep it out of shared configs until every consumer runs v1.8.5, and remove it before rolling back. - A memory-kill escalation steps past the sampled peak, so a single escalation can jump several size classes at once (still bounded by the fleet ceiling).
- The keepalive memory report (
RunningOperations.memory) is safe in mixed fleets in both directions: an older worker never reports and the scheduler falls back to the loss-count rule; an older scheduler ignores the field. - Unknown instance names and missing request fields now return
INVALID_ARGUMENTinstead ofINTERNAL, so clients fail fast instead of retrying: a mistyped--remote_instance_nameerrors immediately rather than after Bazel's retry backoff.
⛰️ Features
- Report each running action's memory peak on the keepalive and escalate a lost worker's action that was over its reservation by @amankrx in (#3170) - (5ca5732)
- Escalate memory on the second worker lost under an action by @amankrx in (#3168) - (a51db80)
🐛 Bug Fixes
- Gate the caps dependency to Linux by @corcillo in (#3174) - (1a29204)
- Fix FS issues running as root by @palfrey in (#3166) - (736e9ed)
- Return INVALID_ARGUMENT for unknown instances and missing request fields by @rajputdivyanshu81 in (#3154) - (ab987d0)
🧪 Testing & CI
- Give the service integration tests the moderate timeout by @corcillo in (#3177) - (2ed2fc5)
- Run Bazel 8 compatibility nightly and on demand by @MarcusSorealheis in (#3171) - (6efef3e)
- Add logging to flaky filesystem tests by @palfrey in (#3164) - (433edda)
⚙️ Miscellaneous
- Pricing page rewrite, comparison table, trial request, and ranked type and radius scales by @modernmedici in (#3167) - (af1646c)
⬆️ Bumps & Version Updates
- Re-vendor remote_execution.proto from REAPI v2.12.0 by @rajputdivyanshu81 in (#3163) - (80f952e)
Full Changelog: v1.8.4...v1.8.5