Skip to content

v0.1.3

Latest
Latest

Choose a tag to compare

View all tags
@JadenFiotto-Kaufman JadenFiotto-Kaufman released this 02 Nov 21:02
v0.1.3
f53f6a4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

0.1.3 is a reliability release, with one change to how a model that runs its own code gets deployed. Whether a checkpoint's own modelling code runs is now part of the model key, and a new trust_remote_code user tag says who may deploy such a model. The fixes keep replicas serving after a timeout, a kill or a failed request, keep the controller tracking nodes and replicas when a node dies, and keep the API working through a Redis outage. The default WARM budget drops from 0.9 to 0.7 of a node's RAM.

If you run NDIF with auth on, read the upgrade notes before deploying: the new tag needs a row in Postgres, and the --trusted flag and the request's trusted field are gone.

Highlights

trust_remote_code comes from the model key (#342)

Whether a checkpoint's own code ran at load used to depend on the caller: a trusted request deployed the model with trust_remote_code=True, an untrusted one with False, and nothing in the deployment recorded which. Now the model key says it. A client that builds TransformersModel(..., trust_remote_code=True) sends a key carrying the flag, and NDIF sizes and loads the model with whatever the key says. Two keys that differ only in the flag are two deployments.

  • Who may deploy one. A key with the flag is deployed only for a request whose API key has the new trust_remote_code tag, or trusted, which implies it. Anyone else gets this on their request instead of a load:

    Could not deploy this model. This model runs code from its own repository (trust_remote_code=True), and deploying such a model needs an API key with the trust_remote_code or trusted tag. Ask an NDIF admin to deploy it, or to grant your key the tag.

    The check applies only when nothing is running for that key. Once the model is up, every caller can use it. The tag grants nothing else: its holder's code still runs in the sandbox.

  • Operators set the flag with --trust-remote-code on ndif deploy, scale, evict and restart, with trust_remote_code: in models.yaml, or with the checkbox in the dashboard's deploy and schedule dialogs. ndif status, ndif export and the dashboard's deployment card show it.

  • A vLLM model key has no field for the flag, so a vLLM deployment can't run a checkpoint's own code.

Replicas no longer wedge after a timeout or a kill (#343)

  • Timeouts. When an untrusted request hit its execution timeout, the interrupt meant for it could land on a pooled worker thread after the request had already finished. It killed that thread, and the replica's next request waited on it forever (#284). Each request's block now runs on a thread of its own, so a late interrupt can only reach that request. In a test with a 15 s timeout, the request after a timed-out one used to hang past 60 s; it now returns in 0.4 s.
  • ndif kill now interrupts the block on the actor. Before, it failed the request at the API while the actor ran the block to completion, and the replica could be sent its next request while the old one was still running. The replica now stays in the pool and takes the next request. The user sees Request cancelled by operator. The CLI waits up to 15 s for the kill's reply, up from 5 s.

A failed request no longer leaves GPU state behind (#350)

  • Memory. After a request failed, its exception's traceback held every frame of the block, and with them the forward's tensors and, if the block took gradients, its autograd graph. Cleanup couldn't free them, so the next request ran with them still allocated. On Llama-3.2-1B, a ValueError after a gradient-tracking forward left 1453 MiB allocated, and the same 1024-token request that had just succeeded twice then ran out of memory three times in a row. A failed request now leaves nothing allocated.
  • Multi-GPU replicas. Cleanup waited only for the current card's kernels before freeing memory. If a block raised while a later card still had work queued, which happens on any error during a backward pass, cleanup freed memory those kernels were still using. The next request, from any user, then failed with CUDA error: an illegal memory access was encountered and the replica restarted. On Llama-3.2-3B across two A100s at 2048 tokens, an error inside with loss.backward(): did this in 5 of 5 trials. Cleanup now waits for every card the replica uses, and it happened in 0 of 5.

nnsight 0.8.2 fixes a related leak in nnsight itself: a backward pass that failed partway left its block suspended, holding the graph for the life of the replica (nnsight #793).

Multi-GPU replicas trace again after a trip through the CPU cache (#340)

A replica spread across several GPUs that was moved to the CPU cache and brought back raised OutOfOrderError on every trace until it was restarted (#339). Moving the model back to the GPUs replaced the module forwards that nnsight instruments, so the model ran without them. The restore now puts nnsight's forwards back. It also passes the skip_keys that transformers uses on the first load, without which a restored multi-GPU Gemma-3n or Gemma-4 would have raised KeyError on every forward.

The controller keeps working when a node dies (#351, #353)

  • Node sync. The controller's node sync asked Ray for live nodes with a filter that makes Ray raise whenever any dead node record exists. The first failure ended the sync for the life of the controller: dead nodes stayed registered, and the same machines never registered again when they rejoined. A controller restarted on a head with a dead node record failed to start at all and stayed down until the next redeploy. The sync now filters in Python, and an error is logged and retried on the next tick.
  • Live replicas missing from /status. /status listed actors with Ray's default limit of 100, which counts a dead record for every actor Ray has run. Past that, live replicas fell off the list. In production, 8 of 47 replicas went missing. The dashboard's reconcile job couldn't see a pinned 70B replica and deployed two more copies. Those filled three 4-GPU nodes, and an unpinned 70B model that had been moved to the CPU cache couldn't come back: its requests failed with CANT_ACCOMMODATE for about 16 hours. /status now asks Ray only for actors that aren't dead, and a replica the controller tracks with no live actor shows as UNHEALTHY.
  • Requests to a replica that is gone now fail in about 0.2 s instead of waiting in QUEUED with no error (#325). The fix to node sync means dead replicas are now actually killed, and a request waiting on one when that happened could wait forever.

Redis outages and dead connections (#355, #357)

  • Dispatcher. Two of the API dispatcher's loops cleaned up in Redis inside their error handlers. During a Redis outage that cleanup raised too and ended the loop, so /status and /env stayed broken until the API restarted, even after Redis came back. The other two loops retried with no pause, 3,692 and 5,558 times a second. All four now survive an outage, pause 1 s after a failed pass, and recover once Redis is back.
  • Results lost after the work was done. The sync Redis client had no retries. A model actor writes its result through it, so one dead connection failed a finished request with Error submitting request to model deployment. The sync client now retries three times with exponential backoff, the same as redis-py's default. The async clients don't retry, on purpose: a subscriber that reconnected without telling anyone would miss what was published meanwhile and wait forever instead of failing.

Upgrade notes

  • With auth on, add the tag before granting it. Insert its row into Postgres, then add it to keys:

    INSERT INTO user_tags (name, description)
VALUES ('trust_remote_code', 'may deploy checkpoints that run their own code');

    Keys with trusted need nothing.

  • --trusted is removed from ndif deploy and ndif scale, and trusted: from models.yaml. Use --trust-remote-code or trust_remote_code: for a checkpoint that ships its own code. A models.yaml that still says trusted: true loads, and the field is ignored.

  • trusted is replaced by trust_remote_code in DeploymentConfig, in each deployment in /status, and in the dashboard's POST /api/deployments/deploy. There it defaults to false; trusted defaulted to true.

  • The request's trusted and priority fields are replaced by tags. Only a client that set them by hand is affected; nnsight never sends them. With auth off, a client opts into the sandbox by sending tags: [].

  • The default NDIF_MODEL_CACHE_PERCENTAGE is 0.7, down from 0.9 (#356). The WARM budget counts only parked weights, at their padded size. On 251 GiB A40 nodes, padded sizes ran 1.14 to 1.21 times what the weights used, and everything else on the node took 38 to 64 GiB, so a full budget at 0.9 went past the 95% of RAM where Ray's memory monitor starts killing workers. A killed WARM replica reloads onto the GPU without the controller knowing, and a model the controller then places on that GPU can run out of memory. At 0.7 a node holds fewer WARM replicas, so some restores become reloads from disk. The overhead is closer to fixed than proportional, so nodes with less RAM may need a lower value.

  • nnsight is no longer pinned (#359). The server uses block_scope from nnsight 0.8.2, and pip doesn't upgrade an nnsight that's already installed, so upgrade both in an existing environment: pip install -U ndif nnsight. The Docker images install the latest nnsight.

Other fixes and improvements

Model actors

  • CUDA error: unspecified launch failure now restarts the replica, like the two CUDA errors already treated as fatal (#354). It leaves the CUDA context broken, and in production a 405B replica failed every request with it for about 8.5 hours until it was restarted by hand.
  • The model actor, tensor-parallel shards and sandbox runner use nnsight's block_scope instead of NDIF's copy of it (#359). Behaviour is unchanged.

Dashboard

  • A dropped Ray client no longer takes down the dashboard's host (#352). The next Ray call started a local Ray cluster on the host instead of raising. Three of them in 20 s used up a 3.8 GB host's memory, and it stopped answering for 40 minutes. start.sh now sets RAY_ENABLE_AUTO_CONNECT=0. The reconcile cron job runs under flock -n, so a tick that finds a run in progress exits instead of queuing, and under a 30-minute timeout, so a hung run ends.
  • The monitor records its connectivity datapoint, applies the up/down transition and sends its Discord message before checking models (#352). A recovery tick whose model check ran past the 480 s script timeout used to exit having recorded nothing, so no "back up" message was sent and the next tick saw a recovery again.
  • The monitor says why a /status check failed: the exception's type and message, plus the response body for an HTTP error. The message starts /status failed: instead of /status unreachable, since a 503 or 504 means the API did answer (#349). Its timeout goes from 30 s to 60 s, the API's own wait for a status refresh. The Discord webhook and the report job's InfluxDB client share that timeout.
  • The monitor can check vision-language models. The image now installs torchvision alongside torch, from the same index. Without it, Qwen3-VL's processor failed to load and every check of such a model failed before reaching the API (#338).
  • The daily report posts to its own discord_report_webhook, falling back to discord_webhook, and its formatting is cleaner.

Docs

  • The add-a-GPU-node runbook said a request running on a node that goes away is re-queued. It usually fails, the same as on a replica restart (#353).

What's Changed

Full Changelog: v0.1.2...v0.1.3

Contributors

JadenFiotto-Kaufman
Assets 2
Loading