Skip to content

Releases

Ratchet 0.5.1: ran on Nebius, built for judges

Choose a tag to compare

@harshbhx harshbhx released this 29 Oct 16:16
794ff71

Ratchet 0.5.1 is the hackathon submission release. The gateway code is 0.5.0; this release adds what ran on Nebius, with dated evidence, and a site built for judges to check it themselves.

Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was retrained on Nebius AI Cloud, and the student was served from a Nebius Serverless Endpoint in a measured test.

What ran on Nebius

  • Retrained on Nebius AI Cloud (27 Oct 2026). The round 2 text-to-SQL recipe, as one Serverless Job on a spot L40S: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%), about $2.16. It reproduces. Evidence: evidence/text2sql-round2-nebius. Model: ratchet-nemotron-nano-4b-text2sql-nebius.
  • Served from a Nebius Serverless Endpoint (27 to 28 Oct 2026). 16 users at once: median 1.6 s (text-to-SQL) and 1.2 s (tool calling); cold start about 7.2 to 7.6 minutes; deleted afterwards. Evidence: evidence/student-endpoint-nebius.
  • Night School end to end on Nebius (28 Oct 2026). Gateway on a CPU VM, training on a spot L40S (966 steps), the golden gate on a Serverless Endpoint. The gate rejected the candidate at 81.8% precision (9 of 11) against a 95% target, so it was never made live. The run's BOM is signed with Ed25519 key 81b0a086250a81d0. Evidence: evidence/nightschool-nebius.
  • Token Factory prices checked (29 Oct 2026) against the price list it serves. Evidence: evidence/day1.

For judges

  • JUDGES.md and ratchet.dualmindlab.tech/judges: a five-minute path, dated proof cards and a verify command for each claim.
  • Judge mode: a judges' API key (shared privately) switches on one-click tests on /judges and lifts the visitor limits on the live request box.
  • All-time numbers: the live site, /live and the dashboard show cumulative totals that survive restarts, a separate server-errors count, and dated measured runs alongside. No bare zeros.

Live site

  • Cloudflare Turnstile human check, verified once per visit and shown right under the button; keyless curl keeps a small daily allowance.
  • The live split is stated correctly everywhere: Nemotron 3 Ultra teaches, and a random 40% of eligible requests go to the hosted Nemotron 3 Nano 30B, a quarter of them re-checked by Ultra. Ratchet's own adapter is not connected to the live site yet.

Docs

  • A glossary of 139 terms with hover definitions across the guides.
  • "Ran on Nebius" and "Ready, not switched on yet" boxes replace the old status notes.

Honest limits

Ratchet's own adapter goes live only after it passes the gate; Night School runs by hand today; the one-command Nebius host is ready but has not run as one command; no savings on live traffic are measured yet.

Built by Harsh Bhanushali.

Ratchet 0.5.0

Choose a tag to compare

@harshbhx harshbhx released this 25 Oct 18:57
e470cfb

Ratchet is an OpenAI-compatible gateway for license-clean distillation on open models. A small Nemotron 3 Nano 4B student with a LoRA adapter answers first; Nemotron 3 Ultra on Nebius Token Factory is the teacher. Every number below can be recomputed from the saved answers in evidence/ (see docs/VERIFY.md).

Results (measured, strict scoring)

Text-to-SQL (headline)

Untrained Teacher (Ultra) Adapter
Exam 1 (172 questions) 12.8% 25.6% 39.0%
Exam 2 (200 questions, new industries) 12.5% 15.0% 33.0%

At the 95% precision bar the student answers 1 of 86 (exam 1) and 0 of 100 (exam 2) questions by itself, so most traffic still goes to the teacher.

Tool calling (second result)

Untrained Adapter Teacher (Ultra)
Hard exam, 150 questions, 20 unseen tools 62.7% 90.7% 91.3%
BFCL, 300 questions 77.3% 80.3% 88.3% (adapter is lower than Ultra)

Adapters (SHA-256 of adapter_model.safetensors)

Adapter SHA-256
Text-to-SQL, round 1 a883bcf6bbf4152efb5a874520b8fa1a09d9bead55c172507725dc255f18236a
Text-to-SQL, round 2 38f99d441e0527b75fdf8fd9d10acbf4473dcf70db7febe3ad73fccd631a6aa7
Tool calling, round 3 fde47de3913fdf14d29328a174b5a92411c3f058989017c7709b4c572db324b9

What is real, and what is not

  • Adapters were trained on a borrowed GPU, on synthetic data labelled by Nemotron 3 Ultra on Nebius Token Factory (OpenMDW-1.1).
  • Teacher calls run on Nebius Token Factory. The live demo and the simulation run on Cloudflare.
  • The Nebius AI Cloud path (Serverless Endpoint for the student, Serverless Job for Night School, a CPU VM and Object Storage) is scripted and tested offline, ready to run with make nebius-plan and make nebius-up.
  • Simulation numbers are labelled Simulation.
  • Not an official product of Nebius or NVIDIA, and not endorsed by either.

Try it

What is new in 0.5.0

See the changelog: live and simulation demos side by side, a waking-up state for cold starts, Scout example questions and a runs-left counter, a one-command Nebius path (make nebius-plan), evidence, link and secret checks in CI, and post-deploy smoke tests.

License: Apache-2.0 for the code. Model weights and data keep their own licenses.

Hackathon build, 24 Oct 2026

Pre-release

Choose a tag to compare

@harshbhx harshbhx released this 24 Oct 11:58

The state of Ratchet at the hackathon submission, as a pre-release.

Real (measured on real models, recomputable): two text-to-SQL exams; the round 2 adapter scores 39.0% and 33.0% strict, the untrained model 12.8% and 12.5%. The data, adapters and every answer are on Hugging Face; docs/VERIFY.md lists the command that checks each claim.

Live demo: https://live.ratchet.dualmindlab.tech makes real Nebius Token Factory calls under spend limits. The teacher answers every request: no trained student is served yet. Three recorded live Scout runs (real Token Factory and Tavily calls) are in evidence/scout.

Simulation: the cascade, release gate and rollback demo run on mock models and are labelled as simulation: https://demo.ratchet.dualmindlab.tech.

Not run: training and serving the student on Nebius Serverless Jobs and Endpoints (scripted and tested offline). The adapters were trained on rented T4 GPUs.

Code under Apache-2.0. Model weights keep their own licenses (NVIDIA Nemotron Open Model License for the Nano base, OpenMDW-1.1 for Nemotron 3 Ultra).