The historic version: NeuroBrix reads, sees, hears, SPEAKS, DRAWS and codes. Three generative omni models — speech out, image out, conditional-flow-matching voice out — each validated on all four execution modes and published on the hub.
Added
- Conditional-flow-matching voice synthesis for multimodal builds
whose speech contract declares a CFM token2wav chain, on ALL FOUR
execution modes: the language model's answer conditions a
codec-token autoregressive stage (additive text-embedding +
normalized hidden-projection merge), then a conformer flow encoder,
a guided cosine-grid flow-matching decoder and a neural vocoder —
all packaged components, with per-speaker conditioning constants
precomputed into the package. Both engines share the seeded draw
AND initial-noise frontiers, so the codec tokens are byte-coupled
across all four modes; verified end-to-end by independent
speech-to-text on every mode's WAV. Adds theNBX_SPEECH_DUMP
diagnostic (default-off) capturing the leg's bisection points
(span ids, speech codes, mel, waveform) on both engines. - Generative image output (
--mode image) for multimodal builds
that declare an image-generation contract, on ALL FOUR execution
modes: the language model conditions a flow-matching diffusion stack
that lives entirely inside the package (learned query tokens as a
package constant, condition bridge, denoiser and VAE as traced
components), with guided sampling through the internal CFG protocol
and scheduler factory. Both engines draw the initial latents from
ONE shared seeded gaussian frontier — bit-identical noise, so runs
are byte-reproducible per engine and cross-engine differences can
only come from kernel numerics. Builds without the contract refuse
cleanly. - Generative speech output (
--mode audio) for multimodal builds
that declare a speech contract, on ALL FOUR execution modes: speaker
selection, seeded reproducible sampling (one shared draw path — each
engine's audio output is byte-reproducible run to run, and the two
engines stay draw-for-draw coupled until a genuinely flat sampled
distribution makes the pick engine-sensitive), and WAV output at the
model's declared sample rate. Builds without the speech contract
refuse cleanly with an explanatory message. - Per-draw speech sampling diagnostics (
NBX_DRAW_DIAG): log the
top-4 candidates, the top-2 margin and the chosen code at every
speech sampling step, on both engines through the one shared draw
path — the adjudication tool for cross-engine code divergences. - Per-step decode diagnostics (
NBX_DECODE_TOPK): both execution
engines can log the top-4 next-token logits and the top-2 margin at
every decode step — the measuring tool for adjudicating rare
greedy-decoding token flips between engines. - Reproducible speech seed from the package: speech-enabled builds
now carry their default RNG seed in the package defaults; the CLI
--set global.seedoverrides it per request (seed 0 is a valid
value). Packages without a declared seed refuse with a clear message
instead of silently falling back.
Changed
- The speech leg executes every codec-embedding lookup and the
per-step residual-code head projection as graph components (a
single-step predictor component invoked once per residual code
group), removing all raw weight access from the flow layer. Both
execution engines run the same graphs by construction; the sampling
frontier is unchanged. Requires a package built with the matching
speech contract — older speech packages refuse cleanly at the
contract check, as before.
Fixed
- Half-precision creation-op fill sentinels:
full-class ops that
carry a mask sentinel next to an explicitly recorded dtype no longer
overflow when the hardware dtype policy narrows that dtype — the
scalar now clamps to the resolved target dtype's finite range
(numerically inert for attention-mask sentinels), on BOTH the
compiled and the sequential execution paths through one shared rule. - Sequential-mode resample ops traced from scale factors: upsample
ops recorded with a null output size and float scale factors now
recompute the size from the live input length (mirroring the
compiled wrapper), and float scale factors in op argument lists are
no longer integer-coerced by the shape resolver (a 1/480 downsample
scale became 0, crashing the vocoder source resample). - Pinned-noise reproducibility on the compiled path: the shared
seeded RNG pin (NBX_FORCE_RAND_SEED) now also covers compiled
execution — all execution modes draw stochastic vocoder noise from
one shared stream, making pinned runs byte-reproducible per engine. - ATen alias ops in Triton modes: numpy-style ATen aliases
(multiply,divide,subtract, comparison aliases) recorded by
vendor code now canonicalize to their base ops in one shared map —
kernel dispatch, op classification and precision rules all treat an
alias exactly like its canonical op. - Text-mode answers on speech-enabled multimodal builds: builds whose
language-model graph exposes an auxiliary mid-stack output (used by the
generative-speech leg) produced garbled text in text-only requests —
the runtime picked the auxiliary output as the final hidden states by
position. Hidden-state extraction now prefers the output the build
NAMES as principal (both execution engines); text answers are restored
byte-identically to their pre-speech baselines. - Ming-Lite-Omni decode is now deterministic (greedy) as the vendor
intends: the packaged sampling default made every run stochastic;
with the corrected package, all four execution modes produce identical
answers token-for-token on text, image, and audio inputs. - Corrupted vertical band on short-schedule video generation (Triton
engine, V100): the flash-attention kernel's known instability on
pre-Ampere GPUs turned out to be structurally biased, corrupting a
fixed window of tokens that classifier-free guidance then amplified
into a visible band. Affected attention shapes now take the
deterministic attention path on V100-class hardware (hardware-profile
driven): the band is gone, the same video renders byte-identically
across repeated runs for the first time, and generation is faster on
the affected configuration. - Serving large-vocabulary models no longer fails at warmup with a
spurious GPU out-of-memory error (Triton engine): the softmax kernel
used at sampling loaded the whole vocabulary row in one register tile;
at 150k-token vocabularies the kernel launch itself silently required
~2 GiB of GPU memory for register spill, and could fail on a busy
device even with gigabytes reported free. Vocabulary-scale rows now use
a bounded-tile streaming softmax whose launch needs no extra memory at
any vocabulary size; smaller rows keep the existing kernel unchanged. - Warm serving now actually keeps weights loaded between requests:
the persistent-serving flag was never forwarded to the per-component
executors, so the daemon silently unloaded the full weight set after
its warmup and re-loaded it during the first request — on large
multi-GPU models (57 GB MoE sharded across 4 GPUs) that mid-request
reload could crash the request with a GPU illegal-access error, and
in all cases defeated the purpose of warm serving. The daemon now
serves warm requests without any reload, which also makes the first
response dramatically faster. - Cross-GPU tensor transfers in the fused MoE path now use the shared
synchronized transfer helper: the previous local copy skipped the
source-device synchronization barrier, leaving a latent read-ordering
race on multi-GPU expert placements.