Skip to content

fix(scheduler): gate fetch flake jobs on fetch capability + adaptive fetch/eval split - #299

Merged
DerDennisOP merged 10 commits into
mainfrom
fix/252-fetch-capability-gating
Jun 1, 2026
Merged

fix(scheduler): gate fetch flake jobs on fetch capability + adaptive fetch/eval split#299
DerDennisOP merged 10 commits into
mainfrom
fix/252-fetch-capability-gating

Conversation

@DerDennisOP

@DerDennisOP DerDennisOP commented Jun 1, 2026

Copy link
Copy Markdown
Member

Part 1 — Fix #252: SSH fetch failing (root cause: job assignment)

authentication required but no callback set; class=Ssh was a job-assignment bug, not an SSH bug (as the reporter suspected):

  1. Every repository FlakeJob is dispatched with a Repository source + FetchFlake task, which requires the fetch capability.
  2. The scheduler's job_eligible_for_caps returned true for all eval jobs — never checking the worker's fetch capability — so a worker with only eval+build could be assigned the clone job.
  3. send_credentials_for_job correctly sends the SSH key only to fetch-capable workers, so the non-fetch worker got has_ssh_key=false.
  4. clone_and_checkout registers a git2 credentials callback only when a key is present → libgit2 fails with exactly that error.

Fix: gate flake-job eligibility on the fetch capability when the job carries FetchFlake, symmetric with credential delivery. WorkerBuildCapsWorkerCaps carries the fetch flag from the negotiated GradientCapabilities.

Fixes #252.

Part 2 — Adaptive fetch/eval split

Builds on the capability gate to free scarce fetch workers. When an idle dedicated eval-only worker is connected, a flake eval is split:

  • a fetch-only job ([FetchFlake], Repository) runs on a fetch worker, archives the source and pushes its NAR to the cache;
  • on completion the server reads evaluation.flake_source and enqueues a cached eval follow-up ([EvaluateFlake, EvaluateDerivations], FlakeSource::Cached) that any eval worker runs, substituting the source NAR from the cache first.

When no idle eval-only worker exists, the original bundled fetch+eval job is dispatched unchanged. A ReserveFetchWorkersRule scoring penalty steers fetch workers away from cached-eval jobs (a soft steer, not a ban — no starvation).

Tests

TDD throughout (cargo test -p scheduler --lib, -p worker --lib):

  • worker_pool: idle_eval_only_worker_detected, draining_eval_only_worker_does_not_count
  • jobs: fetch_flake_job_requires_fetch_capability, cached_eval_job_runs_without_fetch_capability, is_fetch_only_*, cached_followup_rewrites_source_and_tasks
  • scheduler_tests: fetch_only_completion_enqueues_cached_eval_followup
  • policy: reserve_rule_penalizes_fetch_worker_for_cached_eval_only
  • worker eval: cached_source_requires_store_path_present

Docs updated: docs/src/tests.md, docs/src/scheduler.md.

Follow-up ideas (not in this PR)

  • Gate the split on a fetch worker also being present (has_idle_eval_only_worker() && has_fetch_worker()).
  • warn! when a fetch-only job's source NAR push fails (it's load-bearing for the follow-up's substitution).

Every repository FlakeJob carries a FetchFlake task and clones its source
(over SSH for private repos), but the scheduler offered these jobs to any
worker. A worker without the `fetch` capability would receive the job, get
no SSH credentials (the server only sends them to fetch-capable workers),
and fail the clone with "authentication required but no callback set".

Eligibility now requires `fetch` for flake jobs carrying FetchFlake,
symmetric with credential delivery. Eval-only follow-up jobs (cached
source) remain servable by any worker.

Fixes #252
@DerDennisOP DerDennisOP changed the title fix(scheduler): gate fetch flake jobs on worker fetch capability fix(scheduler): gate fetch flake jobs on fetch capability + adaptive fetch/eval split Jun 1, 2026
@DerDennisOP

Copy link
Copy Markdown
Member Author

/gradient run checks.x86_64-linux.gradient-cache

@DerDennisOP
DerDennisOP merged commit 6c175f2 into main Jun 1, 2026
9 checks passed
@DerDennisOP
DerDennisOP deleted the fix/252-fetch-capability-gating branch June 1, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fetching via SSH fails

1 participant