Live Relearn eval image: implement the harvest contract so cortex can pin a digest - #2
Merged
Merged
Conversation
The image is now the whole live scorer for challenge relearn. It reads the harvest request the control plane stages on stdin, scores the submitted artifact against the holdout that request carries, writes the one-line metrics sidecar the harvest reads back, and prints RELEARN_METRICS= followed by RELEARN_EVAL_OK only after the document has passed the control plane's own acceptance checks. The document is RelearnEvalMetrics: the BaselineMeasurement envelope plus the run identity, so a pod cannot answer with another artifact's numbers or replay an earlier run, and the same output is installable as RELEARN_BASE_CHAMPION_FILE. Series are measured, never derived: holdout, perturbed, and public are judged by the frozen teacher over HTTP, canaries and the general-bench canary are graded against shipped references, agent trace against an ordered rubric, and each vision family in the holdout gets a pixel-shuffle control. The deleted harness computed its numbers from sha256(digest || seed); nothing here can produce a score without the model and the judge, and every failure path ends the run with no document and no completion marker. No secret, teacher endpoint, or holdout item is in the tree, and no Modal. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Covers the four properties the control plane depends on: the markers and the one-line sidecar encoding, identity binding to the requested run, refusal of another run's document (wrong artifact, replayed submission, other image, other holdout, missing series, leaky series keys), and refusal of silence (no document, no completion marker, marker without a document). Also pins the commitment port against the control plane's properties, the fail-closed paths of the entrypoint, stdout hygiene (no holdout prompt leaves the pod), and source-level guards that keep an offline scorer, a fallback judge, a baked endpoint, or a forbidden dependency out of the build. The runner and judge doubles live in the test tree so they cannot be installed into the image. Co-authored-by: Mathis <echobt@users.noreply.github.com>
lint-and-test runs ruff and the contract tests. image-contract builds the image and drives it the way the harvest does — request on stdin, relearn-eval score, markers read back — asserting that a pod with no judge, or one handed a tampered request, refuses without printing a document or a completion marker. publish-eval-image pushes ghcr.io/<owner>/relearn-eval and reports the pushed sha256 digest for the control plane's eval_image_digest pin. The contract-only image publishes as a separate job from the scoring image so a runtime build that exhausts runner disk cannot take the digest down with it. Co-authored-by: Mathis <echobt@users.noreply.github.com>
docs/EVAL-IMAGE.md documents the run, the request and every refusal, the document shape, the operator environment, what the image will not do, and how to build and pin it. The README and miner docs still named Qwen/Qwen3.8-Flash-Next, zai-org/GLM-5.3 as the teacher, and Inferact/GLM-5.3-NVFP4. The live pins are Qwen/Qwen3.8-27B, wire id glm-5.3, and incoai/GLM-5.3-NVFP4 served from RELEARN_TEACHER_LOCAL_DIR. Co-authored-by: Mathis <echobt@users.noreply.github.com>
A registry repository must be lowercase and the owner is CortexLM, so buildx refused every tag. Resolve the image name per job instead of interpolating the owner directly, and drop the layer cache from the runtime build: those wheels are past what the Actions cache holds, and a failed cache export would fail a build that had already produced a pinnable digest. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The dtype keyword was renamed in transformers (torch_dtype -> dtype), so the transformers backend now tries both: the image has to load on whichever version the pod's base layer ships. The auto backend also falls back to transformers when vLLM fails to initialise for any reason, not only when it is absent, since auto means whichever backend this pod can actually load. Adds unit coverage for backend selection, weight resolution, adapter detection, and the decode width, all of which are reachable without a GPU. Co-authored-by: Mathis <echobt@users.noreply.github.com>
…rs has The published image resolves transformers 5.x, where the image-text-to-text auto class is no longer AutoModelForVision2Seq and the dtype keyword was renamed. The base model is a native VLM, so picking the wrong class means loading it as text-only or not at all: try AutoModelForImageTextToText, then the 4.x name, then the language-model class, and fail the run when none of them can take the weights. Also documents the one-time GHCR visibility step: a package a workflow creates starts private, and a Lium pod boots with no registry credential. Co-authored-by: Mathis <echobt@users.noreply.github.com>
echobt
marked this pull request as ready for review
August 31, 2026 18:03
The pinned image came up RUNNING and returned no RELEARN_EVAL_OK, so the champion baseline was never recorded. Two causes, both addressed here. The run could not finish. A 120-item holdout is around 350 generations and 280 judge calls once every slice is counted, and the image did them one at a time, 512 tokens wide, after pulling the base model inside the harvest's run timeout. Now: the judge is probed and the weights are checked for presence before the model is loaded; downloading the base is refused unless RELEARN_ALLOW_MODEL_DOWNLOAD is set, because spending the whole timeout on tens of gibibytes is how a pod returns nothing; generation is batched and judging is concurrent, with results kept positional so concurrency cannot put a score on the wrong item; and slices ask for the width they need, a letter rather than 512 tokens for multiple choice. The failures were unreadable. Every dependency now names itself in one line (no judge, no model, artifact digest mismatch), SIGTERM and SIGINT print the phase and elapsed time instead of stopping mid-transcript, an optional RELEARN_RUN_BUDGET_SECS stops between phases, and the libraries that log a line per HTTP request no longer bury all of that in the 8 KiB the harvest keeps. Also adds , so a pod can be checked without spending a run on it. Markers still go to the process's own stdout as well as the sidecar. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Loads a real (tiny) transformers model and answers a real judge over a socket, then runs the entrypoint as a process the way the pod does: score must exit 0, write a one-line metrics.json, print both markers on its own stdout, and produce a document verify accepts — for a champion-baseline request with no miner artifact. The failure cases are asserted the same way, because a 503 with no explanation is what the live run produced: an unreachable judge, weights the pod does not have, a SIGTERM mid-holdout, and a spent budget each exit non-zero, write no sidecar, print no marker, and leave exactly one stated reason. Skipped where the model runtime is absent; a CI job installs a CPU torch so it runs on every change. Unit coverage for batching, judge concurrency ordering, phases, and the budget runs everywhere. Co-authored-by: Mathis <echobt@users.noreply.github.com>
… it cannot Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live 127 on sha256:00839671: the pod came RUNNING, SSH PATH was /usr/bin:/bin, and `relearn-eval` was No such file. `ln -sf /usr/local/bin/relearn-eval /usr/bin/relearn-eval` is a dangling symlink when pip put the console script somewhere else (CUDA / conda / venv), and worse, it overwrites a working /usr/bin script pip already installed. install-cli.sh now writes a regular file at /usr/bin/relearn-eval that execs the interpreter that can import the package, by absolute path, and proves `env -i PATH=/usr/bin:/bin relearn-eval --help` and `score --help` inside the image build. The entrypoint execs that same binary. CI repeats the harvest-PATH check against the built image. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live 127 on sha256:86240d76: timeout could not exec /usr/bin/relearn-eval. Harvest-PATH CI only ran against the slim contract image, and the scoring job reported a pin without pulling the digest it just pushed. Add eval/Dockerfile.scoring (CUDA Ubuntu base) that COPY-installs eval/bin/relearn-eval as a regular file. After push, the scoring job pulls THAT digest and runs the harvest-PATH check; failure means no pin. Do not pin 86240d76 again. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live harvest of sha256:201cc5d2 found /usr/bin/relearn-eval, then failed: `no judge: glm-5.3 did not answer: teacher reply carried no score`. GLM-5.3 thinking is mandatory. max_tokens=32 was spent on the think block, so vLLM returned HTTP 200, finish_reason=length, content=null, and the tokens in `reasoning`. str(None) became "None", which is not a score. Raise the judge budget to 1024 (RELEARN_TEACHER_MAX_TOKENS), send reasoning_effort=low, and extract a non-empty content / reasoning / reasoning_content string. Never str(None). Empty replies name finish_reason. Do not send enable_thinking=false. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live harvest of sha256:cbc4bbb8 exited 1 before the 27B load: `ImportError: Qwen3VLVideoProcessor requires Torchvision`, with a transformers fallback because vllm was never installed. runtime extra now includes torchvision>=0.19 (companion floor for torch>=2.4). Dockerfile.scoring pip-installs .[runtime,vllm] and proves `import torchvision; import vllm` at build. No weights, hosts, or secrets are baked in. Judge body is unchanged. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Live harvest of sha256:cbc4bbb8 came up RUNNING, passed preflight, then spent the rented GPU loading and exited 1 on `ImportError: Qwen3VLVideoProcessor requires Torchvision`, after vLLM had already been skipped because it was not installed. Champion not recorded. Both of those imports are knowable in seconds. deps.py names what a run needs, probes it by import (metadata says a wheel was unpacked, not that its extension module loads), and preflight refuses on what this run will actually touch: the vision imports when the holdout carries vision items, vllm when the operator asked for it by name. `auto` may still fall back to transformers, but the transcript now says it did. `relearn-eval selftest` is the same check with nothing optional and no request or judge needed, so it can be run against a pulled digest. Co-authored-by: Mathis <echobt@users.noreply.github.com>
The build already imports both, but the build is not what a pod boots — the same gap that let sha256:86240d76 ship a /usr/bin/relearn-eval that 127'd, because CI only ever checked the slim contract image. publish-eval-image now pulls the digest it pushed and, before it reports anything pinnable, runs `import vllm, torchvision` on those bytes and `relearn-eval selftest` through PATH=/usr/bin:/bin, which covers the launcher selecting an interpreter that cannot import them. ci.yml keeps the mechanism honest on every PR without a quarter-hour CUDA build: the contract image must fail selftest and say torchvision and vllm. Dockerfile.scoring selftests on the harvest PATH as its last word, and pins its CUDA base by digest like the slim base already did. Co-authored-by: Mathis <echobt@users.noreply.github.com>
Co-authored-by: Mathis <echobt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the eval image the control plane's live scorer needs, matching
docs/RELEARN.md§ Eval image contract and thecrates/relearn-lium-harvestclient oncortex#201.Digest to pin
Republishing from
68b59b2.publish-eval-imageis building; the digest goes here once that job has pulled it back and proved it can score. Until then, nothing on this branch is pinnable.Do not harvest
sha256:cbc4bbb8, and do not re-rent it. Do not pin these earlier digests either:sha256:cbc4bbb80e421952696156a0ceb6d646448e421770ab01d8ad7be5dc989ae0f5Qwen3VLVideoProcessor requires Torchvision; vllm extra never installedsha256:201cc5d29c219097642d61ce4dd713d482a4d0502e49699a22f0a94da4983aaamax_tokens: 32,contentwas null, client parsedstr(None)sha256:86240d7617d296dc12c9f215b6156b127c60dc2baafe87db6dea7a3b7bbb68ba/usr/bin/relearn-evalENOENT (exit 127)sha256:0083967170ddc0fc892857ef8df12db93ff40e3c62b30ded6459d26df5c9498arelearn-eval: No such file or directory(exit 127)sha256:303c63573c9492b1214fa8071977b26a96acc25eaba258c2958c4773a3cd5e45RELEARN_EVAL_OK(timeout / silence)sha256:d0c62f1ca78ff24f3399145cc73133782cd018139447a69f7a485c7eee875944Before the pin works: make
ghcr.io/cortexlm/relearn-evalpublic (or give the pod a pull credential), and prime the pod withRELEARN_BASE_MODEL_DIRor a warm Hugging Face cache.This revision: prove the scoring runtime where it counts
553b557puttorchvisionin theruntimeextra and madeeval/Dockerfile.scoringinstall.[runtime,vllm], and the build of that commit did resolve torch 2.13.0, torchvision 0.28.0, transformers 5.16.1 and vllm 0.28.0. The hole that was left is the one that also produced the 127 onsha256:86240d76: the build is not what a pod boots, and nothing checked the pushed bytes.relearn-eval selftest— no request, no judge, so it runs against a pulled digest. Imports the whole scoring set and exits 2 naming what is missing. A contract-only build fails it on purpose.publish-eval-imagepulls the digest it pushed and, before it reports anything pinnable, runspython -c 'import vllm, torchvision'on those bytes andrelearn-eval selftestthroughPATH=/usr/bin:/bin, which also covers the launcher picking an interpreter that cannot import them.preflightimports the runtime this run will touch before the model loads: the vision imports when the holdout carries vision items, vllm whenRELEARN_EVAL_BACKEND=vllm.automay still fall back to transformers, but now the transcript says it did.cbc4bbb8would have refused in seconds instead of exiting 1 minutes into the 27B load.ci.ymlkeeps the mechanism honest on every PR without a 15-minute CUDA build: the contract image must failselftestand must name torchvision and vllm.eval/Dockerfile.scoringselftests on the harvest PATH as its last word, and pins its CUDA base by digest (nvidia/cuda:12.8.1-runtime-ubuntu24.04@sha256:ebef3c17…) so the bytes under a reviewed digest cannot change without the review.Judge body is unchanged:
max_tokens1024,reasoning_effort"low", noenable_thinking.RELEARN_EVAL_OKis still printed last, only after the document passes the control plane's own acceptance checks, and only on ascorethat exits 0.The contract
request.jsoninto/tmp/relearn_evalover stdinrelearn-eval score --request - …relearn-eval score --request request.json --out metrics.json/usr/bin/relearn-eval(regular file); sidecar is one line, no trailing newlineRELEARN_METRICS=<document>thenRELEARN_EVAL_OKEntrypoint and
serveare unchanged, and the pin is still a digest, never a tag. Document isRelearnEvalMetrics(flattenedBaselineMeasurement+ run identity). No host, secret, Modal, or DFlash2 is baked in.