v1.3.0

[1.3.0] - 2026-10-23

Upgrading from 1.2.0. Follow INSTALL: the first update runs the 1.2.0
updater, so this release's packaged Studio file check starts with the next one.
If a host's Workers run from a release slot, use only dgxm update --verify
(TROUBLESHOOTING #118). The torchmonarch pin and the required dependencies do
not change. Studio needs the studio extra, which neither update mode installs
(INSTALL). Releases before this one refuse a cluster.toml that holds
[studio], so add the table only after the driver and every checkout that
reads the file run this release (Studio). MOSS-Image and Kromoss need ComfyUI 0.42.0 or newer on the
driver and every Worker (TROUBLESHOOTING #125). A graph that wires Model
Sampling SD3 (DGX Monarch) to a checkpoint that samples at multiplier 1.0 now
refuses where it rendered noise (TROUBLESHOOTING #128).

Everyday use

• Add an optional Studio companion, dgxm-studio, with bundled workflow
  controls, durable job records, project libraries, reviewed imports, grid
  experiments and job-bound evidence. It is a draft; read the
  acceptance record first.
• dgxm studio starts Studio with one word. A new [studio] table in
  cluster.toml names the driver origin, the port, an optional driver service
  and an optional start command (CLUSTER); --driver and --config
  override it. It passes the config file dgxm itself uses, prints where each
  value came from and never probes ports. The first start on a fresh state
  directory needs [studio] driver or --driver, and Studio reads only a
  cluster.toml no other account can write (Studio).
• Studio opens signed in. On a terminal with a display, dgxm studio and
  dgxm-studio send the browser to sign-in.html, an owner-only page in the
  state directory that carries the link http://127.0.0.1:<port>/#token=...,
  when .html files and http links open in the same browser. The page keeps
  the token in the tab's session storage and removes it from the address bar;
  the browser's history keeps the link, so treat it like a password. The token
  never appears in a process argument dgxm starts, a query string, a log or
  the journal. --no-open starts no browser, and a companion started with
  --allow-management makes no link; its token is typed into the form.
• dgxm studio tunnel <host> prints the one SSH line that reaches Studio from
  another computer,
  ssh -o ExitOnForwardFailure=yes -L 127.0.0.1:<port>:127.0.0.1:<port> <host>,
  the address to browse and what to do when ssh cannot forward the port. The
  line names the address the forward listens on, so ssh exits instead of
  opening a session without the forward; --run runs it (#133).
• When no driver accepts a connection, Studio shows Driver not running with the
  command that starts it, disables Review generation and connects by itself
  when the driver answers (#131). The README front page shows Studio,
  captured against the CPU test fixture.
• Studio's Review generation takes empty text in a required input, as ComfyUI
  does. It had blocked 23 of the 71 bundled workflows as shipped, among them
  the MOSS-Image ones, which leave double_layers empty, and those with an
  empty negative prompt. A missing value or an unavailable choice still blocks
  review, and an Operations field the companion requires still asks for a value.
• A library import that the companion refuses with record_window_limit,
  because the catalog grew past the browser window after the review, clears
  the pending import, so file selection and a new review work again. The page
  had kept the import pending, and each retry resent the refused plan until it
  expired or the catalog shrank.
• dgxm top and the ComfyUI sidebar show memory in GiB, draw the memory bar on
  hosts with no slab-resident model, give the six 16-color segments six colors
  and show RX and TX rails for every box (TUI).
• Add five MOSS-Image workflows, for 71 in all.

Installation and updates

• Verified update, release staging and setup refuse a target whose packaged
  Studio files are missing, extra or differ from their recorded SHA-256 table.
  Studio's own update needs --config and --driver-service and defaults to
  verified mode. The owner token prints only to a terminal, otherwise to a 0600
  file (Studio, #123).
• The artifact manifest pins the six MOSS-Image DiT files, the BF16 and INT8
  ConvRot encoders and the FLUX.2 small decoder at Comfy-Org's and lodestones'
  commits, and the two Kromoss LoRAs.
• dgxm studio up installs and starts dgxm-studio.service, a user unit with a
  fixed port that also starts the [studio] driver_service unit when one is
  set. dgxm studio down removes it and keeps the unit when its stop fails;
  dgxm studio open prints the sign-in link, opens it when the session has a
  display and says when the service runs code older than the checkout;
  dgxm studio token prints the token file and, with --rotate, restarts the
  service for a new token. dgxm uninstall refuses while the service is
  installed (Studio, #132).

Compatibility and verification

Node mappings, ordered inputs and defaults stay compatible; Skip Layer Guidance
(DGX Monarch) is a new node. The consent route records an optional source,
panel (the default) or studio, as the waiver row's consent_source and
refuses any other value with a 400; Studio's accept sends studio
(threat model). The new consent source moves the Gate protocol to 14, which
burns 13, so with the new version each combination re-proves once on first use
(TRUST, TROUBLESHOOTING #114). The MOSS-Image record
is the release's acceptance record; on 2026-10-22 its identity legs ran again
on ComfyUI a071826fd and matched (VALIDATION). On 2026-10-21 a Studio lifecycle pass on
two Sparks ran Recycle, a respawn render and Fleet under the observer lease,
then dgxm restart, with no failure event (Studio acceptance). The
merged pull requests hold the detail, and TROUBLESHOOTING numbers refer to
that file.

Rendering

• Support MOSS-Image, the OpenMOSS team's 5B single-stream text-to-image
  model, with its SFT and Base checkpoints and lodestones' Kromoss fine-tune
  (MODELS). It needs ComfyUI 0.42.0 or newer, the first release with
  ComfyUI's support commit e087d8f6c (TROUBLESHOOTING #125). The moss_image
  adapter shards the joint [image, text] stream in one contiguous window per
  rank, keeps the prelude and both text-embedder blocks whole on every rank,
  runs each block's fused-norm calls at stock's row count, names pad rows on
  every topology and refuses a padded ring class K. Under
  sequence parallelism it honors only an identity or sequence_shard_local
  block replacement, such as ComfyUI's Skip Layer Guidance skip, and refuses
  any other class P. Five BF16 templates ship at the vendor's settings: 50
  Euler steps at CFG 3 with the layer-22 skip for SFT, Base and Kromoss, a JSON
  layout graph on SFT, and 8 steps at CFG 1 for Kromoss with its DMD2 LoRA.
  Nine testing graphs cover the INT8 ConvRot files, an odd grid, a 1536 square
  canvas, the skip turned off, the second DMD LoRA, the DMD2 LoRA on Base and a
  LoRA slot.
• Promote MOSS-Image SFT, MOSS-Image Base and Kromoss to HW on the hardware
  ladder of 2026-10-21 at ff8dab8300aa with ComfyUI d52a7a5ef, re-measured
  on 2026-10-21 and 2026-10-22 at 9caa6aae660b, the head capped at 2100 MHz
  (VALIDATION). BF16 uly2 was bit-identical to one GPU at one step and over
  50 steps from 1024 to 2048 square on SFT and at 1024 and 1536 square on Base
  and Kromoss, and on SFT to stock at one step from 512 to 2048 square; auto
  keeps it (BENCHMARKS). FSDP renders matched resident ones, so FSDP is
  supported. cfg2 keeps refusing on the shipped graphs, which carry the skip,
  and gets no row; ring2 is measured, not granted. The Kromoss DMD LoRAs bake to
  stock's merge on every key; their renders differ from stock's because stock
  applies a LoRA at cast time.
• The four CFG 3 MOSS-Image templates use ComfyUI's normal scheduler, within
  0.0005 of the vendor pipeline's sigmas at 50 steps where simple sits 0.069
  off; on 2026-10-21 the SFT graph on normal rendered on uly2 bit-identical to
  one GPU and to stock. The 8-step Kromoss graph and the testing graphs keep
  ...

dgx-monarch 1.2.0

[1.2.0] - 2026-10-16

Upgrading from 1.1.0. Follow INSTALL: the first update runs the 1.1.0
updater, so this release's checks start with the next one. If a host's Workers
run from a release slot (remote hosts of guided multi-node setups, every host
after a verified update), use only dgxm update --verify; plain update leaves
them on 1.1.0 (TROUBLESHOOTING #118). From 1.2.0 on, plain update refuses such a
host and Doctor's worker release version row fails a stale one. Before
updating, turn rdma_latent_return off, add no [monarch] table and clear
DGXM_RDMA_QP_SPLIT where Workers start. Then install xxhash and what
pip check names on every host, clear what TROUBLESHOOTING #119 names, add a
[monarch.rdma] target before re-enabling native return, rerun the editable
install after a verified update, and restart ComfyUI. The torchmonarch pin moves
to 0.7.0; FLUX.3 Klein needs ComfyUI 0.41.0 or newer, or commit 20b46065e.

Everyday use

• A ComfyUI that stays up attaches again after its Workers restart within the
  30 s delivery timeout, and refuses older Workers before it dials (closes #2).
  So does a sampler render that meets restarted Workers, recording reattach
  and reattach_s (closes #47); the two-Spark recheck ran on 2026-10-16.
• Eager setup and Fleet queues heal an abandoned lease without Reset attached
  mesh. The sidebar shows the topology and each host's rail rates, red when a
  rail is down, and progress no longer sticks after a render.
• Add three Kroma v0.3.1 Turbo workflows, which matched stock at batch 1, and
  four FLUX.3 Image Klein workflows, for 66 in all.

Installation and updates

• From 1.2.0 on, plain dgxm update refuses a release-slot host before it
  fetches (release_slot_worker, release_slot_unknown), checks the fetched
  commit's dependencies on every host, fast-forwards only if they pass, and
  refuses an uncommitted pyproject.toml edit or a linked worktree.
• Doctor, dgxm up and dgxm restart read a managed Worker's torchmonarch from
  its release; the skew warning sends a slot to --verify; Doctor checks that
  /usr/bin/python3 is 3.9 or newer on each host.
• Verified updates take --privileged-process-inspection, keep exact legacy
  Worker units, and accept a foreground inspector of up to four hours.
• Every manifest row pins a Hugging Face commit, size and sha256 or names the
  file you supply (101 pinned, 6 supplied), and CI checks it. The LTX 2.5
  Gemma 4 row pins the vendor's current file, checked again against stock.
  setup_env.sh --comfy no longer needs torchaudio.

Compatibility and verification

Node mappings, ordered inputs and defaults stay compatible. The new pin and
version make each combination re-run its first-use Gate (TROUBLESHOOTING #114);
later edits that cannot change math, residency or transport keep PASS rows.
Migration checks ran the shipped Krea2 workflow at 1536 square on that build's
uly2 route (row 11; slab Gate PASS) and the shipped H3 workflow (output only) on
ComfyUI e8665ca. Later records ran on ComfyUI 29e28e2b4 (0.41.0); the
FLUX.3 Klein record is the release's acceptance record. The merged pull requests
hold the detail, and TROUBLESHOOTING numbers refer to that file.

Rendering

• Support FLUX.3 Image Klein, step-distilled and base, for text to image and
  reference editing (MODELS, TROUBLESHOOTING #121). Both rows are HW. uly2
  matched one GPU exactly on text to image from 1024 to 1536 square and, since
  each fused norm runs at stock's row count, on edits and small sizes
  (2026-10-16). ring2 and base cfg2 are measured, not exact; NVFP4 refuses FSDP.
• auto takes cfg2 where it was exact and beat uly2 in Campaign K: Krea2 FP8
  and BF16 from 1.2 to 2.36 MP, Chroma BF16, FP8 and INT8 to 1.64 MP, Qwen-Image
  FP8 to 1.77 MP, Boogu FP8 and Anima BF16 under 1.2 MP, and CFG++ at CFG 1.
• A checkpoint or LoRA that pins an approximate attention kernel refuses class K
  where the pin runs, until you grant the Sol-Attn waiver (TROUBLESHOOTING #117,
  closes #30). Sequence parallelism drops the pin. A BF16 or FP16 checkpoint's
  own pins refuse class P on +fsdp, and first_use refuses a running pin on
  cfg2+fsdp and dp2+fsdp even with the waiver.
• Refuse a checkpoint whose weight scales ComfyUI would drop, such as FLUX.3
  Klein's fp8r file, class P with no waiver (TROUBLESHOOTING #120).
• A Gate FAIL quarantines only its own combination, LoRA keys on FSDP-replicated
  modules bake, Anima runs sequence-parallel again, padded Ulysses ran 2.2%
  faster in a Chroma test, and FSDP reload pricing credits freed shards.

Operations

• Add a validated [monarch] policy for startup, RDMA and diagnostics. Config
  loading names a bad key, native return needs a nic: target per host, a
  policy edit reads POLICY_STALE (TROUBLESHOOTING #113) instead of locking out
  lifecycle commands, and a refused edit keeps the fleet and its export.
• A mesh build or Worker start refuses an ambient OTLP endpoint with export off
  and MONARCH_RDMA_* or DGXM_RDMA_QP_SPLIT values the policy did not set
  (TROUBLESHOOTING #119); so does an export that cannot start.
• Add opt-in Monarch metrics with ownership occupancy, OTLP export, and phase
  timings that split each native return. /dgxm/metrics adds step time,
  native-return latch and faults, and per-port rails; the finish record adds
  topology, table row, quant, kernel and pins. Monarch's profilers stay off.
• A failed native read latches native return off for the session
  (TROUBLESHOOTING #115). Doctor adds listener generation and worker policy
  rows and, like dgxm restart, names why a source probe failed. Fix partial
  dgxm diagnostics captures, dgxm top rates and phases, sidebar refresh
  after a reset, a Reset that raced a loader preflight and stale ledger temp
  files. Load profiles live in ~/.cache/dgx-monarch/profiles/.

Development

• CI type-checks under check_untyped_defs; canaries cover pins, scale replay,
  LoRA bakes and local actors. The sweep labels pinned cells and CFG 1 cfg cells
  under FSDP as refusals; the matrix runner records waivers and refusal classes.
  tools/make_pin_probe_lora.py and rdma_return_bench.py --timing are new.

Limits

• Native latent RDMA stays off by default; selected-rail checks keep their
  recorded scope. Not yet run on hardware: FSDP LoRAs on H3, LTXAV or Anima, the
  credited Flux2 proof, return timing, clean cancellation and Kroma's batch-2 dp2+fsdp.
• Kroma keeps Ulysses: its cfg2 failed the same-math bar on 2026-10-16, so Krea2's cfg2 row excludes it.

DGX Monarch 1.1.0 — Onboarding and operator improvements

DGX Monarch 1.1.0 brings the onboarding and operator improvements added since 1.0.0 into a compatible source release.

What is new

• Normal-terminal access through opt-in dgxm launcher, preserving existing commands and supporting safe reruns.
• ComfyUI and its owned terminal dashboard launch together with --with-top, including from a desktop shortcut.
• Separate health and activity: healthy work shows READY / RENDERING, with a wait instruction for maintenance.
• Optional fleet-wide Sol-Attention planning, installation and diagnostics, plus an explicit GPU kernel smoke test.
• A MiniMax H3 workflow that saves combined H.264/AAC MP4 at 24 fps. The catalog now includes 59 workflows.
• Clearer guided setup and update diagnostics, distribution-metadata Torch constraints, and support for verified existing package layouts.

Existing node mappings, ordered inputs and defaults remain compatible. Core runtime dependency pins are unchanged. See the changelog.

Installation and updating

Follow the source installation and update guide, preserving the working CUDA Torch build. Older updaters that refuse a recognized package layout have a documented recovery procedure. First-use identity checks must run again after an upgrade before cached results can be reused.

Verification

Release commit: 9f6408b4e15356da99300e4cd4fa52d30740d458.

Final-commit hosted CI and the ComfyUI compatibility canary passed. The identical candidate tree passed 11,017 local CPU tests (29 skipped), 756 extracted-source tests (1 skipped), static/generated-surface checks, local package checks and a disposable editable source install. The source-install smoke was repeated successfully from the public v1.1.0 tag; its reviewed plan changed only dgx-monarch, and recorded dependency metadata and core-file stat bindings remained unchanged.

The exact release commit completed verified deployment, Doctor, a two-rank cluster smoke and actor-source readback on both Sparks. Worker services were restored running and idle. The runtime source differs from the preceding hardware-tested commit only in the package version string; the full Sol and MiniMax checks below were not repeated on this release commit.

The dated operator acceptance identifies the earlier hardware-tested source, 09957d971441c59d4605092c064886c66841eb6e. Both GB10 GPUs passed the Sol CuTe dense kernel control. A two-rank TORCH_FLASH MiniMax run produced a decoded H.264/AAC MP4, and the TUI displayed healthy rendering separately from idle. The run carried an existing accuracy waiver. Its identity gate had no LoRA/slab configuration to compare and remained INCONCLUSIVE; this is functional evidence, not an exact-fidelity comparison. Source-install checks used disposable environments with explicitly reused dependencies, not clean-machine provisioning.

Limits

Native latent RDMA remains NOT RUN / HOLD. Model support remains scoped to the checkpoint, precision and topology recorded in the model guide. Sol's dense kernel smoke does not establish sparse-attention or model fidelity; approximate render use requires its explicit accuracy waiver.

The cuSPARSELt 0.8.1 ARM64 packaging exception remains necessary and issue #5 stays open. Use the maintained dependency checker; do not replace a working Torch build to remove that diagnostic.

This is a GitHub source release. Wheels and source packages were built locally for validation and are not published as release assets. Runtime dependencies and model files are installed separately under their own terms.

DGX Monarch 1.0.0

DGX Monarch brings distributed image and video generation to a pair of NVIDIA DGX Sparks in ComfyUI.

Included in 1.0

• Sequence and CFG parallelism for supported renders, Fleet for independent prompts, and FSDP for supported models that need more capacity.
• 58 example workflows, including Krea2, Ideogram4, MiniMax H3 and Qwen Image 2.1.
• Guided setup, Doctor, verified updates, a terminal dashboard and a ComfyUI sidebar.
• A copyable coding-agent prompt and one maintained setup skill, with a Chroma first-render recipe and verified model-file metadata.
• Installation, model, benchmark and troubleshooting guides. Test-only graphs live with the test fixtures.

Start with the installation guide, first render, or coding-agent prompt.

Verification

The setup procedure completed a fresh-agent trial with new Python environments, ComfyUI and Monarch on two existing Sparks. Two distributed Chroma renders produced saved images, and a setup rerun preserved the installation. The trial reused the OS, drivers and verified model files; it was not a clean-machine or cold-cache test.

The release runtime at 4647189 completed Doctor, a two-rank cluster smoke check and a saved distributed Chroma render. Both Sparks participated. The client mesh was recycled, the Workers remained healthy, and temporary test firewall rules were removed with the original rules verified. The runtime source is unchanged in the final release commit, 14dacda. Subsequent changes cover documentation, repository links, tests, CI and the dependency checker.

Hosted CI passed on x64 with Python 3.11 and 3.12 and on ARM64 with Python 3.12. The ComfyUI compatibility canary, local package-content checks and disposable CLI installation checks also passed for the final commit. Those local installations reused verified dependencies. Both Spark checkouts were synced and passed Doctor without restarting their Workers. See tested configurations for the scope of individual hardware results.

Limits

Support depends on the checkpoint, precision, inputs and parallel mode. Model support and benchmarks include the settings and limitations. FSDP provides capacity and can be slower.

Native latent RDMA remains disabled pending stable selected-rail support and the required checks. Latents return through actor messages; NCCL rendering across the pair remains available. The inter-node fabric must meet the documented isolation requirements.

The official NVIDIA cuSPARSELt 0.8.1 ARM64 wheel has a platform-tag defect reported by pip 24.2 and later. Run python tools/check_dependencies.py as described in the installation guide. It accepts this single known error only after verifying the installed files against the official wheel and its published hash. Every other dependency error still fails. The checker passed on both Sparks and in hosted ARM64 CI. Issue #5 stays open until the pinned PyTorch build moves to cuSPARSELt 0.9.0 or later and a fresh ARM64 environment passes the required checks.

Other remaining work stays open: native latent RDMA, Worker recovery, and the temporary xFuser compatibility build.

This is a GitHub source release. Runtime dependencies and model files are installed separately under their own terms.