CapyCTL 0.1.7

CapyCTL 0.1.7

CapyCTL 0.1.7 stops work nobody is waiting for. When a client hangs up, on a streamed
or a non-streamed request, CapyCTL now stops the engine's work on it, and on SGLang it
aborts that one request and stops counting it as soon as the engine ends it. Each model
instance on vLLM 0.32.0 or SGLang 0.5.22 can get its own private LMCache, a KV cache in
host RAM, so a prompt that begins like an earlier one is answered faster, across a park
too. Engines no longer report usage off the host by default. vLLM 0.32.0 is supported for
chat, a discrete GPU's startup peak is measured instead of estimated, and failed SGLang
launches fail fast and clean up after themselves.

Requests

• A client that hangs up on a non-streamed request now stops the engine's work on
  it. Until now only a streamed answer was stopped when its client went away; a
  non-streamed chat request or a decision request kept the engine busy until its whole
  answer was ready. CapyCTL now closes the engine connection as it does for streams,
  also for a model served on a host, and counts the request as it counts any cut
  request. See When a request is cut short.
• A streamed request whose client hangs up before its first token is cancelled at
  once. It used to stay open on the engine, and uncounted as cancelling, until the
  engine produced its first token for it, which for a queued request could be long
  after.
• On SGLang, a cancelled request stops counting as soon as the engine ends it. On
  SGLang 0.5.22 with one tokenizer worker, CapyCTL sends each chat request its own
  request ID (the X-CapyCTL-Request-Id value, also the answer's id). When a client
  hangs up, or a request is cut by its deadline or idle bound, CapyCTL asks SGLang to
  abort that request and settles it when SGLang ends it, instead of waiting for the
  engine to read idle, on the router's machine and on hosts alike. When cancelled
  requests waiting for their instance's next stop fill a model's 32 places, CapyCTL
  restarts the instance, which settles them, rather than refusing new requests with
  queue_full.
• A request cut before the engine answered it no longer settles on the engine's idle
  reading. A request whose client hung up, or that missed its deadline, before the
  engine sent any of its answer may not have reached the engine's queue yet, so the
  engine reading idle proved nothing about it. Such a request now stays counted until
  the engine's next stop or restart; stopping and switching do not wait for it, and a
  model that would park stops instead. A request the engine had started answering, or
  answered with an error, settles on the idle reading as before.
  capyctl status deployment shows how many settle each way
  (requests.settle_when_idle, requests.settle_at_stop).
• How quickly each engine stops a request whose client hung up, measured on SGLang,
  vLLM, llama.cpp and TensorFold, is in the
  requests guide.

LMCache

• A private shared KV cache per deployment. On vLLM 0.32.0 and SGLang 0.5.22 or
  newer with LMCache 0.5.6 (or a later 0.5.x) installed and registered, each model
  instance gets its own LMCache server in host RAM, on loopback only, started and
  stopped with it, so a prompt that begins like an earlier one is answered from the
  cache after the engine's own cache dropped it, across a restart_only park too.
  lmcache: auto is the default, but a deployment that parks deep, the default
  residency, runs without LMCache until its engine, model type and GPU are qualified
  for deep parking with it, and this release qualifies none. Declare
  residency: restart_only to use LMCache now (see
  Parking with LMCache).
  Set lmcache: off or required on a deployment, or lmcache: off on a host. The pool (4 GiB by default, lmcache_pool) and the LMCache
  server's own memory are reserved. cache_salt now reaches the engine and LMCache as a
  digest of the deployment, its revision and the salt. See
  LMCache.

Privacy

• Engines no longer report usage off the host by default. Every engine CapyCTL
  starts gets DO_NOT_TRACK=1, and vLLM also VLLM_DO_NOT_TRACK=1 and
  VLLM_NO_USAGE_STATS=1, so vLLM no longer sends its usage report to
  stats.vllm.ai. A host that wants the engines' own defaults sets
  engine_telemetry: allow. A deployment or profile that sets one of these names is
  now refused (engine_env_reserved:<name>). See
  Nothing reported off the host.
• retention: zero never starts LMCache, alongside keeping engine logs in memory
  and turning core dumps off. See
  Zero data retention.

Engines

• vLLM 0.32.0 is supported for chat, deep parking included.

Memory

• On a discrete GPU the startup peak is measured. A first start now measures the
  card and host RAM apart, from the engine's own processes, and later starts set aside
  what was measured instead of the estimate, on each domain once it is measured.
  capyctl status deployment <name> --json says where each figure came from
  (startup.device_provenance, startup.host_provenance). An enrolled host with a
  discrete GPU needs this release to start a deployment whose peak was measured that
  way (host_capability_missing:startup_host_bytes otherwise).

Operations

• Containers run on a discrete GPU. On an RTX 4090 Laptop GPU with Docker and CDI,
  a standalone role running as an unprivileged user started, parked, woke and stopped
  vLLM 0.31.0 and started and stopped TensorFold 0.6.6; per-process GPU memory and the
  card's limits work as on a host. A TensorFold image needs ninja, and a new container
  started on existing state needs the same --hostname as the one before. See
  Running in a container.

Fixes

• A changed hostname is reported clearly. A standalone role started on existing
  state under another hostname, as a new container gets unless docker run --hostname
  is given, used to fail with an opaque resource policy revision conflict. It now
  refuses with host_name_changed (exit 2), naming the stored and the current name and
  the fix: docker run --hostname <stored name>.
• Hosts running under a systemd user service or a template unit were refused. A
  host whose memory is bounded by a cgroup named with @ (user@1000.service,
  capyctl@host.service) reported it, and the server ended its session with
  host session authorization failed. Such hosts now join.
• An SGLang launch whose scheduler dies while starting now fails within seconds. It
  used to wait out the whole readiness deadline and then report that ownership of the
  engine was uncertain. CapyCTL now ends the engine's remaining processes, reports the
  launch as failed with the engine's own log lines, and releases its memory once they
  are gone. The repeated prometheus FileNotFoundError messages that followed such a
  failure no longer appear.
• A failed SGLang launch no longer leaves shared memory behind. A launch whose
  scheduler died while loading left SGLang's /dev/shm/sglang_loads_*.shm file, and
  other shared-memory segments its processes made, in RAM. CapyCTL now removes the
  segments the launch made, once its processes are verified gone and no other process
  holds them.
• A failed SGLang launch's log ends with CapyCTL's failure line. On Python 3.12,
  five lines about ResourceTracker and ChildProcessError used to follow it.

Known limits

• On SGLang 0.5.22, a decision request whose client hung up still runs to its end in
  the engine
  (sgl-project/sglang#45842);
  CapyCTL keeps it counted until the next stop.
• Decision deployments on SGLang run without LMCache for now: SGLang fails with it
  while its radix cache is disabled
  (sgl-project/sglang#45929).
• Decision models (api: decisions) stay on SGLang 0.5.22 and llama.cpp 0.7.0. vLLM
  0.32.0 answers them from their text output instead of their decision head, so CapyCTL
  refuses vLLM for them with capability_missing:decisions
  (Decision models).

CapyCTL 0.1.6

CapyCTL 0.1.6

CapyCTL 0.1.6 runs any engine you choose. capyctl engine add --custom registers
a program CapyCTL has no integration for, from a manifest that says how to start it and
which API it serves; CapyCTL starts it on a private loopback port, refuses it if it
listens anywhere else, and asks you to confirm the installation, which it then pins.
Every answer now carries a request ID, yours too if you send one, and decision answers
carry timing and token metrics. CapyCTL can keep no request data on disk: engine logs can
stay in memory, vLLM starts with request logging off, and a new data retention page lists
what is kept and how to turn it off. Three accounting bugs found by a formal review are
fixed. SGLang's options that dump to disk are reserved, and capyctl engine flags shows
which options CapyCTL owns. A role can run in a container, qualified on a GB10, and a new
guide shows how to drive CapyCTL from a controller program.

Engines

• SGLang options that dump to disk are refused. --watermark-config,
  the --debug-tensor-dump-* options, --msprobe-dump-config,
  --export-metrics-to-file, --export-metrics-to-file-dir and
  --spec-trace-dir are reserved, whatever the host approves, and so are the
  DUMPER_*, SGLANG_DUMP*, SGLANG_KERNEL_API_* and
  SGLANG_CUDA_COREDUMP* variables and a few other SGLang dump variables.
  capyctl engine flags <family> prints which options and variables CapyCTL
  owns, reserves, gates behind an approval or passes unchanged, and why
  (--format json for scripts).
• Custom engines. capyctl engine add --custom <program> --name <name> --manifest <file> runs a program CapyCTL has no integration for, such as a
  model's own server, from a manifest that states its command, the API it
  serves (chat or decisions) and optionally a health path and an idle signal.
  CapyCTL owns the process, gives it a private loopback port, routes and admits
  requests, counts the memory you state in resources and keeps its log; it
  checks none of the program's options, does not park it, and stops a start
  whose engine listens off loopback. The engine is saved in
  custom-engines.yaml beside engines.yaml; an older CapyCTL ignores that
  file. See Custom engines.
• Cancelled requests on an engine with no idle signal. A request a client
  abandons on a custom engine without idle keeps its memory held until the
  engine's next stop or restart, since nothing shows that the engine stopped
  working on it. Stopping or switching models does not wait for such requests,
  and capyctl status deployment shows how many there are and when they
  clear.

Requests

• Request IDs. Every answer from /v1/chat/completions and /v1/systemone
  carries X-CapyCTL-Request-Id, CapyCTL's ID for the request, also written in
  its logs. Send your own X-Request-Id (1 to 128 printable ASCII characters,
  no spaces) and it comes back on the answer and appears beside CapyCTL's ID in
  the router's logs; it is never sent to the engine or stored. Decision answers
  now carry capyctl.metrics too: total time, time to the engine's answer,
  input tokens, and the engine's own figures when it reports them. See
  Request IDs and metrics.

Decision models

• The readiness question is published. Deploy a model
  shows the exact request CapyCTL sends to prove a decision model ready, and
  what a ready answer must contain.

Operations

• Engine logs can stay in memory. Set engine_logs: memory on a host (or
  host.engine_logs: memory in a standalone document) and CapyCTL writes no
  engine log file. It keeps the last 4 MiB of each launch's redacted output in
  memory for the management engine-log read and for the reason a failed launch
  gives, until a minute after the engine exits. --debug-engine-logs is refused
  with it. vLLM now starts with request and output logging explicitly off on
  every supported version, and neither can be turned back on. A new
  data retention page lists what CapyCTL
  keeps about requests, for how long, and how to turn each part off.
• Driving CapyCTL from a controller. A new guide page shows how a program
  uses the management API to create, start, watch, replace and delete
  deployments: the admin token, the snapshot, idempotency keys and retries,
  revisions, the event stream, request IDs, which errors to retry, and how to
  leave every start to the controller. See
  Driving CapyCTL from a controller.
• Running in a container. A host or standalone role can run as the main
  process of a container. On a GB10 with Docker and CDI, a standalone role
  running as an unprivileged user started, parked, woke and stopped a
  llama.cpp decision model, SGLang 0.5.22 and vLLM 0.31.0, and measured each
  engine's GPU memory as it does on a host. CapyCTL publishes no image; the
  guide shows what to put in one. See
  Running in a container.

Fixes

• A park that is cancelled or refused no longer reopens dispatch that a
  controller restart closed. The instance serves again only after its
  readiness is proven in the new session.
• A resources: block whose ready phase claims a device its cold,
  parking or wake phase does not is refused when the deployment is
  written, instead of putting a second exclusive owner on the device when the
  instance becomes ready. A deployment stored before the upgrade with such a
  block keeps running and can be stopped; replace it with a block whose
  larger phases claim the device too.
• A request whose client disconnects while it is being admitted no
  longer stays counted against its instance until the controller restarts,
  where it refused parks of the instance and held switch drains to their timeout.

Known issues

• Custom engines run restart only, on one machine. A custom engine cannot
  park and cannot span machines.
• A custom engine without an idle signal settles cancelled requests only at
  its next stop. Without idle in the manifest, a request a client abandons
  keeps its memory held until the engine's next stop or restart.
• Containers are qualified on a GB10 only. A role in a container has run
  live on a GB10 (unified memory) with Docker. A discrete GPU, Podman, and a
  host joined to a server have not yet been run live in a container.
• basal answers every failure with HTTP 422. In the guide's worked
  example, basal-serve 2.2.0 answers a malformed request and an engine failure
  (out of memory, an internal error) alike with 422, so CapyCTL passes either on
  as a rejected request. Read the end of the engine log to tell them apart.
  The issue is reported upstream.
• basal's fp8 mode needs a package the tested environment lacks.
  --mode fp8 needs torchao; the example uses --mode fast, which takes many
  minutes to start cold, so deployments on it set timeouts.initialize.
• llama.cpp can starve a decision request when several run at once. With 4 or 8
  concurrent decision requests on llama.cpp v0.7.0, a few waited until every other request
  had finished; llama-server alone does the same, without CapyCTL. Set
  engine_config: {max_concurrent_requests: 1} there, or use SGLang when several
  clients share one decision model.
• vLLM has no decision support yet. A decisions deployment on vLLM is refused
  capability_missing:decisions.
• A standalone request record trades durability for latency. On a standalone
  machine a power loss can lose the last request records, which describe engines
  the power loss also stopped.

CapyCTL 0.1.5

CapyCTL 0.1.5

CapyCTL 0.1.5 adds decision models. A deployment with api: decisions runs a decision
model on llama.cpp v0.7.0 or SGLang 0.5.22, becomes ready by answering a fixed decision,
and answers POST /v1/systemone with probabilities; CapyCTL's recipes repository has
measured recipes and a comparison of six decision models. SGLang 0.5.22 now launches, an
engine's error reaches the client as 502 engine_error instead of locking the model,
CapyCTL adds less time to each request, and a large vLLM model that parks no longer gets
an impossible startup estimate before its first start.

Decision models

• Deployments declare their API. A deployment takes api: decisions to
  serve a decision model, or api: chat (the default). A decisions
  deployment runs on llama.cpp 0.7.0 or newer or SGLang 0.5.22 or newer
  (anything else is refused capability_missing:decisions), on one machine,
  without parking and without a tool-call or reasoning parser; a machine
  whose CapyCTL is too old is never chosen for it
  (host_capability_missing:decisions). See
  Decision models.
• Decision requests. A decision model (api: decisions) answers
  POST /v1/systemone on the inference port, with the same key, waiting and
  wake-on-request as chat: typed questions (noul, choice, score) about a
  state, answered with probabilities and the engine's usage. Sending a
  model's requests to the endpoint it does not answer is 400 api_mismatch,
  naming the right one, without waking anything, and GET /v1/models shows
  each model's API as capyctl.api (Decision requests).
• Decision models become ready on a decision. A deployment with
  api: decisions is proven ready by one fixed question it must answer with
  a complete set of probabilities, at start and whenever CapyCTL re-checks
  it; it is never sent a chat request. See
  Decision models.
• Measured recipes. CapyCTL's recipes repository has decision-model recipes for
  Clef-Flash, Clef, Kev-4B and Julia-1 on llama.cpp v0.7.0 and Clef-Flash and Clef on
  SGLang 0.5.22, and a comparison of six decision models on one GB10: speed, throughput
  and quality on five public test sets. On llama.cpp, set --batch-size and
  --ubatch-size to the context; the guide shows
  how.

Engines

• SGLang 0.5.22 launches. SGLang 0.5.22 installs plugins of its own
  (from smg-grpc-servicer, which it requires), and every launch failed
  plugin_closure_failed. CapyCTL now starts SGLang with
  SGLANG_PLUGINS=capyctl-loads-no-plugins, a selection that names no plugin,
  so SGLang loads none, and accepts those installed plugins under it. A
  plugin from any other package is still refused. SGLANG_PLUGINS,
  SGLANG_PLATFORM and SMG_* are now names CapyCTL owns
  (engine_env_reserved:<name>).

Requests

• Lower router overhead per request. The router no longer reads a
  deployment's instances twice per request, reuses its prepared store
  statements, and turns Nagle's algorithm off on its listeners. On a local
  reproduction this removed about a quarter of what the router added to a
  decision request; the request ledger's guarantees are unchanged.
• Less time added per request. A completed answer now reaches the client
  before CapyCTL records the request as finished, and on a standalone machine
  the request ledger no longer waits for the disk on every request. On a local
  reproduction this cut what CapyCTL adds to a decision request from about
  4.5 ms to about 2 ms. A power loss on a standalone machine can lose the last
  request records, but it stops the engines they describe as well. A client
  that sends its next request at once may briefly find the previous one still
  counted against the deployment's limit.

Fixes

• An engine's error no longer locks a model. When the engine answers a
  request with an error (llama.cpp refusing an input longer than its batch,
  for example), the client now gets 502 engine_error with the engine's own
  message instead of 500 backend completion unverified, and the request
  stops counting against the model's 32 running requests once the engine is
  idle. Before, a few dozen such errors left the model answering every
  request with 429 queue_full until it was deleted. See
  Errors.
• A large vLLM model that parks no longer estimates an impossible startup peak. Before a
  first start is measured, CapyCTL estimated a parking vLLM model's startup peak at 3.6 times
  its weights; Nemotron 3.5 Super VL showed 265 GiB. The estimate now adds at most 32 GiB to
  the weights (112 GiB for Super VL), which still covers every peak measured on a GB10. Models
  under about 12 GiB keep the same estimate. A deployment revision accepted before this
  release keeps its old estimate until its first start is measured or a new revision replaces it.
• SGLang 0.5.22 finds its cutlass package. SGLang 0.5.22 imports
  cutlass from nvidia-cutlass-dsl, which only a .pth hook puts on Python's
  path, and CapyCTL never runs those hooks; serving a dense BF16 model failed
  with No module named 'cutlass'. CapyCTL now adds that one directory inside
  the installation to the engine's path itself, still without running the hook.

Known issues

• Decision models run restart only, on one machine. A deployment with
  api: decisions cannot park and cannot span machines.
• llama.cpp can starve a decision request when several run at once. With 4 or 8
  concurrent decision requests on llama.cpp v0.7.0, a few waited until every other request
  had finished (Clef-Flash: 149 s against a 3.7 s median); llama-server alone does the same,
  without CapyCTL. Concurrent requests add almost no decision throughput on llama.cpp, so
  set engine_config: {max_concurrent_requests: 1} there, or use SGLang when several
  clients share one decision model.
• vLLM has no decision support yet. A decisions deployment on vLLM is refused
  capability_missing:decisions.
• A standalone request ledger trades durability for latency. On a standalone machine
  the ledger commits with SQLite's NORMAL durability, as decided: a power loss can lose
  the last request records, which describe engines the power loss also stopped.

CapyCTL 0.1.4

CapyCTL 0.1.4 parks SGLang groups again: a park or wake now records only memory reports
that are still current, which also fixes the cause of the failed SGLang group wake in
0.1.3. capyctl engine add --allow-remote-code replaces hand edits of engines.yaml for
checkpoints that ship their own code, status shows the startup memory a first start
really sets aside, capyctl version names the commit a release was built from, and
standalone reports the engine's own latency. Nemotron 3.5 Super VL is refused parking on
vLLM, whose wake of it answers wrongly.

Engines

• capyctl engine add --allow-remote-code lets deployments on that engine
  run the code a checkpoint ships. It writes security.trust_remote_code: true
  to engines.yaml, so a deployment with engine_config.trust_remote_code: true
  no longer needs a hand edit and a restart; the refusal without it names the
  option. vLLM and SGLang only: TensorFold and llama.cpp refuse it
  (Engines).
• Nemotron 3.5 Super VL is not parked on vLLM. vLLM 0.31.0 answers wrongly after it parks and
  wakes this model, so CapyCTL refuses deep and host_backed for it; declare
  residency: restart_only.

Multi-node groups

• SGLang groups park and wake again. In 0.1.3 an SGLang group across machines
  restarted instead of parking, after a live group wake failed restore_uncertain. The
  cause was in CapyCTL: a park or wake could record a machine's memory report that had
  become too old to accept, either because the report was read before a slow check of the
  woken model or because the other machine reported later. Every report a park or wake
  records is now recent, for groups and single machines alike, and an SGLang group with no
  residency gets deep again. Rechecked live on two GB10 hosts: five park and wake
  cycles each for an SGLang 0.5.21 and a vLLM 0.31.0 group.

Fixes

• status deployment shows the startup memory a first start really sets aside. A model
  whose estimated startup peak is above the machine's managed limit starts alone and sets
  aside the whole limit; status now shows that limit (provenance whole_host) instead of the
  estimate, which could exceed the machine's memory (265.2 GiB for a 71 GiB checkpoint on a
  121 GiB machine).
• capyctl version shows the commit. capyctl version prints the version
  and the commit the release was built from (--json for a JSON object);
  include it in bug reports. capyctl --version still prints only capyctl <version>.
• Engine latency in standalone. GET /management/v1/metrics/latency and
  capyctl status deployment --json now show the engine's own latency
  distributions (tier: engine, source: engine) for standalone instances
  of vLLM, SGLang and TensorFold (llama.cpp exports none), as they do on a
  server with hosts. Before, standalone showed CapyCTL's router series only.

Known issues

• Some models do not wake on vLLM 0.31.0. gpt-oss still fails to reload its weights
  after a deep park, as on 0.30.0, and CapyCTL refuses to park it. Gemma 4 26B-A4B NVFP4
  answers wrongly after a wake; CapyCTL's wake check stops it instead of serving it.
  Declare residency: restart_only for both.
• vLLM 0.31.0 can crash on a repeated long prompt with Nemotron 3.5 Super VL. With
  prefix caching on (vLLM's default), sending the same 32k or 64k-token document a second
  time crashed the engine with a CUDA illegal memory access; CapyCTL restarted it and
  served the waiting request. Turn prefix caching off for this model
  (--no-enable-prefix-caching), as its recipe does.
• A vLLM wake under lazy is slow. On a GB10 with vLLM 0.30.0 and
  Qwen3.8-27B NVFP4, a deep-parked deployment loading with lazy took about
  2 minutes (123 s) to reload its weights in place on its first wake, against
  23–28 s with eager. Keep the default eager where wake time matters.

CapyCTL 0.1.3

CapyCTL 0.1.3

CapyCTL 0.1.3 runs one model across two machines: vLLM, SGLang and TensorFold
groups at tensor_parallel: 2 are qualified live on two GB10 hosts over a
direct link, with deep parking for vLLM groups. llama.cpp's llama-server
joins as a fourth engine, and TensorFold 0.6.6 and vLLM 0.31.0 are verified.
Memory sizing is more accurate on unified memory and for SGLang hybrid and
speculative models, every chat answer carries per-request engine figures, and
the management API reports live load and engine log tails. A wake that fails
or answers differently now stops the model instead of leaving it stuck.

Engines

• TensorFold 0.6.6 verified. CapyCTL lists TensorFold 0.6.6 as verified
  beside 0.6.0 to 0.6.3 and 0.6.5. Its new --name-priority ID=background
  option is an ordinary extra argument; on the second machine of a two-machine
  group, which CapyCTL starts without --name, it may fail.
• vLLM 0.31.0 verified and launches. The reserved argument check now
  follows the destinations the installed vLLM parser itself defines, so a build
  that moved an option reserves the name it spells: vLLM 0.31.0 moved logging
  into the --logging-config object (--logging-config.<field>, with
  --log-level and --log-config-file as flat overrides), and every spelling
  of it stays reserved beside --uvicorn-log-level — a deployment or host
  cannot set the logging configuration, on 0.30.0 or 0.31.0. vLLM 0.31.0
  answers carry the same per-request figures as 0.30.0; cached_tokens still
  maps from usage.prompt_tokens_details.cached_tokens, which 0.31.0 reports
  only with --enable-prompt-tokens-details passed as an extra argument, and
  the figure is left out when the engine reported none, never shown as zero.
• llama.cpp's llama-server as a fourth engine. Register a llama-server
  built from llama.cpp v0.6.0 with capyctl engine add <path to llama-server>
  and deploy a GGUF model directory with engine: llamacpp. CapyCTL reads the
  GGUF header and derives the memory from the weights and a full-window KV
  cache for each of the deployment's slots (max_concurrent_requests, 4 by
  default). A llama.cpp model does not park: when CapyCTL needs the memory, it
  stops it and starts it again on the next request. Answers carry llama.cpp's
  timings and CapyCTL's per-request figures; tool calls and reasoning come
  from the model's chat template. Every llama.cpp build lists as custom for
  now. See llama.cpp.
• TensorFold's key variable is reserved. TENSORFOLD_API_KEY can no
  longer be set in a profile's or deployment's engine_env, whatever the
  approvals: it turned on TensorFold's own authentication and locked CapyCTL
  out of the engine's health and metrics.
• Prefill time on TensorFold's CUDA engine. A TensorFold answer from a
  CUDA Qwen model now carries the engine's prefill_ms; its time to first
  token and decode rate are measured by the router.

Multi-node groups

Qualified live on two GB10 hosts at tensor_parallel: 2 over their direct
link, with RDMA carrying the group traffic: vLLM 0.30.0 groups pass every row
(bring-up, five deep park and wake cycles with the wake canary, a killed head
or worker, a host agent going away and coming back, and alternating with
another model under --evict); SGLang 0.5.21 groups pass bring-up, failure
and recovery, and restart instead of parking; TensorFold 0.6.5 groups pass
bring-up and failure, restart only. Groups of three or more machines are the
next milestone.

• One model across machines. A deployment with a topology and
  placement.hosts runs one engine rank per machine, head first. vLLM 0.30.0
  and SGLang 0.5.21 run tensor and pipeline parallel on any number of machines
  that divides tensor_parallel × pipeline_parallel; TensorFold 0.6.5 runs
  tensor_parallel: 2 on exactly two machines. Only two machines are
  qualified. See
  One model across machines.
• vLLM groups park deep. A vLLM group's park completes once each host
  proves its own rank released its memory, and each member is then charged
  what it really holds parked. A worker's recorded processes include the rank
  process that holds the GPU, and a member that cannot prove its state fails
  the park within a minute instead of fifteen, with every charge kept until
  the group stop releases it.
• SGLang groups restart instead of parking. An SGLang group across
  machines gets restart_only by default, and a group that states
  residency: deep or host_backed is refused at deploy with
  capability_missing:deep_park and a hint to declare restart_only (see
  Known issues). SGLang on one machine still parks
  (Groups across machines).
• New settings. --peer-address, --rendezvous-ports (default
  25000-25099) and --require-rdma true|false (default false) on hosts,
  each also as a variable and a YAML key, and --group-stall-timeout (default
  120s) on the server (settings).
• Engine variables. capyctl engine add --env and --approve-env, and
  capyctl deploy model --engine-env, each also as a variable
  (Engine environment).
• Status lists each member with its host, rank, state and memory charged,
  and marks every group peer transport unauthenticated. Group peers talk on
  unauthenticated ports: keep group machines on a private direct link
  (Network access).
• Ports a group opens. The guide lists every listener a vLLM, SGLang or
  TensorFold group opens, the address it binds and that none is
  authenticated, with example nftables and ufw rules for the direct link
  (Ports a group opens).
• DP attention ports. With SGLang DP attention, the head's start check
  also covers the handshake port P+13, and a rendezvous port above 65522 is
  refused because SGLang cannot bind the handshake there. The per-rank ports
  SGLang picks at start cannot be checked
  (Ports a group opens).
• TensorFold groups start and become ready. A TensorFold group member
  starts with its installation's libnccl (TF_NCCL_LIB); an installation
  without the nvidia-nccl wheel is refused with nccl_library_missing. CapyCTL
  reads the token ids a TensorFold head returns where TensorFold puts them, so
  the group passes its readiness check.
• A member on an unreachable host reads uncertain. When a member host's
  agent goes offline, its member shows uncertain (group_member_uncertain)
  at once and keeps its memory charge. When the agent comes back with its
  journal, the member shows launched again.
• A killed group worker leaves no engine processes behind. When a worker
  member's engine process died while processes it had started still ran (a
  vLLM Worker_TP1 holding device memory), the host now stops every process
  of the member's launch and the group settles. A worker member whose launch
  was never recorded settles on its host's evidence that every process of that
  launch is gone.

Catalog models that run as groups in this release, on two machines at
tensor_parallel: 2 (not part of the live rows):

Model	SGLang 0.5.21	vLLM 0.30.0	TensorFold 0.6.5
Qwen3.8-Flash-Next	NVFP4	NVFP4 (stock build unverified)	MLX 4-bit only
GLM-5.3-Flash	no (container image only)	unverified	MLX 4-bit, or EXL3 (experimental)

Not yet: DeepSeek V4 Flash 0731, DeepSeek V4 Flash Vision and MiMo-V2.5 have
a working build only as a container image; MiniMax M3, Hy4 Preview and
Nemotron 3 Ultra need three or more machines; Hy3 has no validated command.
Next milestones: container launchers, and a live row on three or more
machines.

Memory

• A model split across machines reserves its share of the weights. A
  deployment with a topology whose memory CapyCTL derives from the weights
  charged every machine the whole checkpoint, so a two-machine
  tensor_parallel: 2 group reserved its weights twice and a model larger than
  one machine could never start. Each machine now reserves the weights divided
  by tensor_parallel x pipeline_parallel, plus the tensors every machine
  keeps whole, read from the checkpoint's safetensors headers (a tenth of the
  weights when they cannot be read), and the KV cache, margin and startup
  placeholder are derived from that share. A 126 GiB checkpoint at
  tensor_parallel: 2 reserves about 64 GiB of weights per machine; with
  CapyCTL's startup placeholder (2.25 times the share) it still needs
  memory.startup on two 128 GB machines. Declared resources and
  single-machine deployments are unchanged. See
  several machines.
• Short resources. A model that restarts instead of parking can state
  its memory as resources: {gpu: 11GiB, ram: 2GiB} instead of five phases:
  CapyCTL reserves both figures while it starts, runs and stops and nothing
  once it is stopped. On a unified-memory machine the pool is charged their
  sum. The five phases are still accepted, and `capyctl ...

CapyCTL 0.1.2

CapyCTL 0.1.2

Engines

• vLLM 0.30.0 is verified beside 0.29.0. Its --enable-scale-out is
  reserved, and --watermark-config and --engram-config need a named host
  approval like every other *-config option. vLLM 0.29 and 0.30 stream a
  reasoning trace as delta.reasoning; CapyCTL now relays it, where before a
  deployment with --reasoning-parser failed its readiness check.
• SGLang 0.5.21 is verified beside 0.5.20. A request to a parked thinking
  model now wakes it on both versions; before, the wake check's few tokens went
  to the thinking trace and the wake was left uncertain.
• TensorFold 0.6.2, 0.6.3 and 0.6.5 are verified beside 0.6.0 and 0.6.1.
  0.6.4 was not checked and shows custom yes. TensorFold 0.6.5's
  --api-key, --api-key-file and --metrics-open are reserved: CapyCTL owns
  authentication, and a key on the engine would lock CapyCTL out of its routes
  and metrics.
• A new guide, Install an engine, walks through
  installing vLLM, SGLang or TensorFold in a uv environment and registering
  it. The engines guide has a first model on each engine, with deployment
  files in docs/examples/.

Memory

• 16 GB cards. A model whose charge reaches the card's managed limit
  starts: memory the card already holds outside CapyCTL (driver reservation,
  display server) now comes out of the device reserve instead of being counted
  twice, which left such a deployment queued. A memory.request declared for
  a card derives its KV cache from the request. Checked with a 9 GB model on a
  16 GB RTX 4090 Laptop GPU on vLLM 0.29 and 0.30 and SGLang 0.5.21.
• Standalone memory limits are settable.
  host.resource_policy.memory.system.managed_limit and free_reserve take
  auto (50 % and 20 %, as before), a size (90GiB) or a percentage (75%),
  in YAML, with --set or with CAPYCTL_SET__.... On a discrete machine they
  set host RAM; card limits stay derived. See "Standalone memory limits" in the
  settings reference.
• TensorFold is capped by its declaration. CapyCTL launches TensorFold
  with TENSORFOLD_CUDA_MEMORY_LIMIT_GB set to the deployment's Ready
  allocation. Before, TensorFold sized itself from the machine's free memory
  and could grow past its declaration under long prompts.
• Kernel builds leave the startup peak alone. A first start of vLLM,
  SGLang or TensorFold that compiles kernels no longer records the compilers'
  memory as the deployment's startup peak, which on a unified-memory host left
  every later start capacity_blocked. Builds inside the engine process
  (Triton, torch.compile workers) are still counted.
• A parked model is charged what it holds. After its first park, CapyCTL
  measures the memory a parked engine keeps and charges that, never below the
  placeholder (1 GiB on a card, 2 GiB unified) and never above its Ready
  charge. capyctl status deployment <name> --json shows it under parked.
• SGLang memory.kv_cache is the KV cache. It now sets SGLang's KV pool,
  as on vLLM. A hybrid model's per-request state is sized beside it for
  max_concurrent_requests, or as many requests as fit, up to 32. An explicit
  memory.request too small for it is refused before launch, naming the
  smallest request that fits; a derived one runs as many requests as fit, and
  status shows the limit. On a discrete GPU, a derived request keeps the state
  inside the KV cache CapyCTL chose, up to half of it, and the fitted context
  is held to the rest; status and the start name the same request. Before, a
  hybrid model could be held to 2 running requests.
• First starts and draft models. The first-start reservation on vLLM and
  SGLang covers CUDA graph capture and the load transient
  (max(request + graphs, weights x 2.25 + margin) for new deployments). A
  draft model's weights and KV layers are counted in the request and in the
  fitted context. vLLM gets --max-num-seqs 32 by default, and the fit for
  gated-delta-net hybrids follows vLLM's own block layout.

Reliability

• An engine is no longer stopped when helper processes it started exit on
  their own, such as the torch inductor compile workers SGLang starts with
  CUDA graphs on. Only the engine's own processes exiting stops it; a stop
  still ends the helpers.
• Long prompts. A 256k-token prompt on TensorFold is bounded by the request
  deadline until its first token, instead of being cut by the 120 s idle bound
  during prefill. SGLang's refusal of a prompt longer than its KV pool now
  reaches the client as 400 engine_rejected with SGLang's message, instead of
  a 500 or a stream with no [DONE]. A request the router cuts for its bounds
  is cancelled on the engine, as a hang-up is.
• Standalone queue bounds (host.resource_policy.queue) are settable, and
  a restart applies changed bounds.
• Refusals. capacity_blocked names the host, the memory domain, what the
  deployment needs, what is free and the limit. A start during a slow stop is
  answered still_stopping (retryable, exit 25) instead of
  reconciliation_required. A launch refusal a retry cannot change (an SGLang
  sizing refusal, checkpoint_mismatch, a missing capability) gives up after
  one attempt. checkpoint_mismatch explains that model.content_fingerprint
  is CapyCTL's digest over the whole model directory, and status prints the
  declared and measured digests.
• --wait. start deployment --wait and deploy --activate --wait end
  at once with exit 13 and the reason when the start gives up, instead of
  waiting out the Initialize window. After a redeploy, --wait waits for the
  previous stop instead of returning still_stopping.
• TensorFold first request. The readiness check waits out the kernels
  TensorFold 0.6.1 and later build on their first request, within the startup
  budget, instead of killing the engine mid-build after 60 s. A build lock left
  by a killed start is removed once no recorded process is alive.
• Engine registrations. On standalone, a registered profile's approvals,
  security.extra_args, env and log_policy are now applied, as on a host.
  Before, standalone dropped them, so an approved drafter path was refused and
  extra_args: denied was not enforced. capyctl engine add takes
  --approve-option and --approve-path.
• capyctl engine remove works with no role running: it edits engines.yaml
  and the role publishes the removal when it starts. A role skips a profile
  for an engine it does not know, with a warning, instead of refusing to start.
• A local model path outside the model store (for example a Hugging Face
  snapshot directory) is measured inside its own root. A foreign listener on a
  leased engine port is refused port_conflict before the launch. A failed
  deployment can be updated with deploy --revision.

Features

• Parsers by model family. For Qwen3, Qwen3.5, Qwen3.6 and Qwen3.8
  checkpoints, CapyCTL picks vLLM's and SGLang's tool-call and reasoning
  parsers from the checkpoint's config.json and chat template, so tool calls
  come back structured and reasoning comes back apart from content with no
  extra_args. Set them with engine_config.vllm|sglang.tool_call_parser and
  reasoning_parser (auto, none or a parser name). A parser in
  extra_args or host-fixed arguments wins over auto. Status shows a
  Parsers line.
• TensorFold runs requests together. engine_config.max_concurrent_requests
  renders TensorFold's --parallel, 8 when undeclared; before, TensorFold
  decoded one request at a time. Status shows Streams. TensorFold deployments
  can turn drafts off with extra_args: [--no-drafts].
• SGLang CUDA graphs are on by default while a model can park, about 75 %
  faster decoding on a 16 GB card. engine_config: {cuda_graphs: false} turns
  them off.
• capyctl init host and init server write YAML to .yaml and .yml
  paths. JSON results are indented on a terminal and stay one line when piped.
  A missing --config file is named in the error. Status splits a startup
  charge into card and host RAM, and shows a pending operation's reason under
  the table. Non-streamed vLLM answers now carry usage.

Packaging

The release archive and the installer no longer ship the guides; the
installer and the systemd units point to
https://edurdias.github.io/capyctl/docs/install/. An upgrade removes the old
share/capyctl/docs.

Upgrade

Replace the binary and restart the roles, the server before its hosts. 0.1.1
state is reused; the server store migrates on first start, so back up each
role's state directory first if you may roll back (see "State and migrations"
in the install guide). Then check these:

• A deployment refused capacity_blocked since a first start that built
  kernels keeps the peak 0.1.1 recorded. Delete it and deploy it again once.
• SGLang with speculative decoding (--speculative-algorithm in its
  arguments) now defaults to restart_only: it is stopped and started instead
  of parked, because a wake would not restore the draft model's weights. A
  deployment that states residency: deep or host_backed with it is refused;
  remove the residency or set restart_only, and deploy it again.
• SGLang hybrid models with a stated memory.request and kv_cache are
  refused when the request cannot hold the per-request state beside the
  KV cache. For example, a 48 GiB request with a 16 GiB KV cache and a
  DFlash2 draft model no longer fits 8 running requests. Raise the request to
  the value the message names, or lower max_concurrent_requests.
• TensorFold deployments now start with --parallel 8 and a memory cap at
  their Ready allocation. A resources block sized for one stream may be
  refused at start with "TensorFold's memory cap cannot hold context_length
  beside its streams"; lower max_concurrent_requests...

CapyCTL 0.1.1

CapyCTL 0.1.1

TensorFold

CapyCTL now runs TensorFold 0.6.0 and 0.6.1 beside vLLM and SGLang. Register an existing
TensorFold venv with capyctl engine add <venv>, or name it as the role's own
engine with --tensorfold-bin (CAPYCTL_TENSORFOLD_BIN, local_engine.tensorfold); CapyCTL checks that the CUDA
build tools TensorFold needs on its first start are on the engine's PATH. With
0.6.1 the CUDA compiler can come from pip (pip install ninja "cuda-toolkit[nvcc,cccl]==13.0.*") instead of a system toolkit.
TensorFold models do not park: CapyCTL stops them when it needs the memory and
starts them again on the next request. A TensorFold deployment states its
resources and context_length. Checked on NVIDIA GB10; discrete GPUs run it
unchecked in this release.

Other changes

• A readiness check now accepts a model that answers with reasoning first.
• New exit code 26, toolchain_missing, from capyctl engine add.
• A client that hangs up in the middle of a streamed answer now stops the
  engine's work on it, for vLLM, SGLang and TensorFold. CapyCTL keeps the
  request counted until the engine reports nothing running or waiting, so a
  park or a switch that follows no longer waits for the whole answer. A
  client that reads nothing for 10 seconds, and a single piece of an answer
  larger than 64 KiB, now end the answer the same way; before, the engine ran
  the answer to its end.
• A model named by its Hugging Face cache directory (snapshots/<rev>) is
  measured even when its files link twice, as recent huggingface_hub
  versions store them. A checkpoint that cannot be measured now says why,
  instead of reporting a digest mismatch.
• capyctl validate config without --host checks a resources block and
  refuses a TensorFold deployment without one, and lists what still needs a
  host.
• A role with no engine starts and says how to add one; capyctl engine remove can remove the last engine.
• capyctl status deployment shows the engine a deployment actually runs, with a note
  when that engine is no longer registered.
• Idle models are stopped or parked only when ready_idle_timeout or
  parked_idle_timeout is set; both are off by default.
• CI is a fast gate; the integration suites run with scripts/ci-local.sh --deep
  before a merge, and tests no longer depend on runner speed.

Upgrade

Replace the binary and restart the roles; 0.1.0 state is reused.

CapyCTL 0.1.0

CapyCTL 0.1.0

The first release of CapyCTL. Control what runs next: run your models on your
own NVIDIA GPUs, and deploy, park, wake and switch them on vLLM and SGLang behind
one OpenAI-compatible endpoint, even when they don't all fit at once. CapyCTL
parks the models nobody is using so their memory is freed, and wakes the one a
request asks for.

Install

curl -fsSL https://edurdias.github.io/capyctl/install.sh | sh
capyctl --version

One binary for Linux on x86-64 or ARM64. The installer checks every download
against the release's SHA256SUMS. Add -s -- --version v0.1.0 to pin this
release. Bring your own vLLM or SGLang environment; CapyCTL does not install
engines or drivers.

What is in 0.1.0

• One machine or several. Run everything on one GPU machine (standalone),
  or run a server with several GPU hosts that join it over mutual TLS.
• vLLM and SGLang. Register an existing engine environment with
  capyctl engine add <venv>; a deployment file needs only name, engine and
  model.
• Parking and switching. Idle models are parked to free GPU memory and
  woken on the next request. On a discrete card a model can park its weights in
  host RAM, which wakes much faster than reloading from disk.
• Unified-memory and discrete GPUs. On a discrete card, CapyCTL counts the
  card's memory apart from host RAM and picks a GPU with room on a machine with
  several. One GPU per model in this release.
• Models from a directory or Hugging Face. Models live in ~/models;
  model: {hf: <org>/<name>} downloads into ~/models/sources.
• OpenAI-compatible API. /v1/models and /v1/chat/completions, with
  streaming and tool calls. The endpoint listens on 0.0.0.0:8443 and requires
  the API key; --listen limits it to one address.
• Settings in one place. Every setting can come from the YAML document, a
  flag or an environment variable; capyctl config show prints each value and
  where it came from.
• systemd units for the standalone, server and host roles.
• Commands print readable text by default: tables for lists, and a summary
  with key-value details for single results. Add --json for scripts. Roles
  print text in a terminal and JSON lines to the journal and log files.

Known limits

• Models that span several GPUs (tensor parallelism) are not supported yet.
• No web UI; everything goes through the CLI and the management API.
• Parking uses the engines' development controls. CapyCTL keeps them on loopback
  behind a per-launch key, but they are not production-hardened; a host can opt
  out with --deep-park off.
• SGLang ModelOpt (NVFP4) checkpoints cannot park yet; deploy them
  restart_only.
• CapyCTL does not terminate TLS. Put a TLS reverse proxy in front before exposing
  the endpoint to the internet.

Upgrading from a release candidate

Three things to know when upgrading:

• The project is now CapyCTL: the binary is capyctl, variables start with
  CAPYCTL_, and the services are capyctl-server, capyctl-host and
  capyctl-standalone. Nothing reads the old mllm names, and state an
  earlier release wrote cannot be reused, so 0.1.0 starts fresh:
  1. Stop every deployment with the old binary (mllm stop deployment <name>),
     so no engine it started keeps running.
  2. Stop and disable the old services: systemctl --user disable --now mllm-server mllm-host mllm-standalone (without --user for a system
     install), then delete their unit files from ~/.config/systemd/user or
     /etc/systemd/system.
  3. Remove the old binary and shared files: ~/.local/bin/mllm and
     ~/.local/share/mllm, or /usr/local/bin/mllm and
     /usr/local/share/mllm. Old state (~/.local/state/mllm,
     ~/.config/mllm, /etc/mllm, /var/lib/mllm) is no longer read; keep
     it for reference or delete it. Downloaded models in ~/models stay
     usable; their download records in ~/models/sources/.mllm do not.
  4. Install 0.1.0 and set it up as a new installation (for a system install,
     create the capyctl service user as in the
     installation guide).
     Upgrade the server and every host together: a 0.1.0 server and an
     earlier host cannot talk to each other.
• The inference endpoint of an existing configuration file does not move: a
  file an earlier release generated keeps 127.0.0.1:8443. New files use
  0.0.0.0:8443 with the API key. Set the bind or start with
  --listen 0.0.0.0:8443 to serve the network.
• On a discrete card, standalone replaces its single memory pool with separate
  host-RAM and GPU domains. It stops the engines the earlier release started
  and re-sizes their deployments; the next request starts each one cold.

Upgrade the server before the hosts. The fixed fallback key capyctl-local is
gone: if the credentials file cannot be read, standalone refuses to start.

Documentation

• Getting started: https://edurdias.github.io/capyctl/
• Operations (services, upgrades, rollback): docs/operations/install.md in
  the release archive.