The observability release. A GPU deployment that silently ran everything on
the CPU used to produce identical rows, identical EXPLAIN tags and nothing
above debug — this release makes that visible, from a counter in the
dashboard to a Prometheus endpoint on every worker.
It does not complete the production-readiness milestone. Ten lines of
Phase 8 remain open; three of them (#28, #45, #46) need a
CUDA device to verify on and one (#29) is what the spill manager's real
wiring waits for.
Added
-
The CPU-fallback counter (#32).
fallback_batchesonOperatorStats
andOperatorSnapshotcounts every batch that took the CPU reference path
although its operator was placed on a device. The TUI Inspector shows
cpu fallback N / M— green at zero, yellow when some batches fell back,
red when all of them did — and each operator logs onewarn!on its first
fallback naming the reason. A silent fallback was previously detectable
only by re-running withRUST_LOG=oxidelake_compute=debug. -
Placement notes in
oxide explain(#32). A node the rule leaves on the
CPU is an ordinary DataFusion operator in the plan, indistinguishable from
one that was never eligible.oxide explainnow prints, under the plan,
every node that was skipped and why —AggregateExec: \avg` is not one of
the aggregates the device kernels implement (sum, count, min, max),the
grouping key `s` is Utf8View; the device kernel groups on Int64`. A plan
that lowered completely prints nothing. -
Spans, a per-query log line and a metrics endpoint (#33). An
oxide.operatorspan per operator and partition carryingoperator,
partition,target(what the plan says) andbackend(what is
executing).OxideSession::collectlogs oneINFOline per query with the
mode, rows, elapsed time and fallback count;oxide sqluses it, so
RUST_LOG=oxidelake_runtime=info oxide sql …prints one line per query.
Behind the newmetricsfeature,oxide-worker --metrics-portand
oxide-scheduler --metrics-portserveoxide_*counters as Prometheus
text. Cluster executors never run the placement rule, so the plan codec
attaches a process-wideTelemetryHubto everyGpu*Execit decodes —
without that a worker's/metricswould describe no work at all. The
endpoint is unauthenticated, like the Ballista ports themselves: bind it on
a private interface. Without the feature--metrics-portis refused, never
ignored. -
--batch-size,--output table|json|csv, worker--backend(#49).
--batch-sizeoverrides the CPU default (8192) and the GPU-target default
(65536) onsql,explainandtui; zero is refused rather than silently
returning nothing.--output jsonwrites an array of objects and
--output csvRFC 4180 with a header — an empty result still prints the
header and[], so a script can tell "no rows" from "the query failed".
oxide-worker --backendmirrorsOXIDE_BACKENDand is applied before the
first task, so it reaches the operators rather than being accepted and
ignored. Every flag and environment variable is now in one README table. -
SessionOptions(#49), withOxideSession::local_with_optionsand
connect_with_options.#[non_exhaustive]with builder methods, so a knob
added later is a minor release.
Changed
-
predictreads its activation from the model file (#47). Breaking
for existing model files. Asafetensorsheader declares
{"__metadata__": {"oxidelake.activation": "relu"}}—relu,gelu,
sigmoid,tanhornone, applied between theLinearlayers and never
after the last. A file that declares nothing is refused with a message
naming the key and the alternative:predict(path, features, 'relu'), a
new third argument that must agree with the header when the file has one.Fixing this found a worse bug. The loader built a stack of bare
Linear
layers and applied no activation at all, while the module docs, the
README and ADR-0015 all said "ReLU between them" — so every non-linear
model returned a linear model's numbers, and the only test of the numbers
compared them against the same bare stack. Thenonecase is now checked
against oxmera's ownSequential::forward, which does not share the
loader's loop, andrelu,geluandnonemust disagree with each other. -
Public enums are
#[non_exhaustive](#41).BackendKind,
Compression,SessionMode,MemoryClass,MemoryTier,Tier,
ModelSpec,Activationand the TUI'sPanel/KeyInput/Transition
now need a_arm in a downstreammatch.ModelSpec::SequentialMlpRelu
is gone, replaced byModelSpec::SequentialMlp { activation }(see
predict, below). The plan vocabulary stays
exhaustive on purpose —Predicate,Comparison,Literal,
AggregateFunction,DistanceMetricand the codec'sGpuNode: every
kernel must handle every variant, so a_arm in the CPU correctness
reference is a wrong answer that compiles.docs/RELEASING.mdrecords the
distinction, and the DataFusion coupling thatcargo-semver-checkscannot
see. -
The plan codec carries a version fingerprint (#42).
postcardis not
self-describing, so aGpuNodefield added or reordered between builds
decodes without complaint into the wrong parameters and a mixed-build fleet
computes a confident wrong answer. Every encoded plan now carries a magic,
a format version and a fingerprint over the crate version and the compute
layer's feature mask; a plan from a different build is refused at decode
with both fingerprints named. A committed snapshot pins the bytes of all
four variants, so a wire-format change cannot land unnoticed. -
Memory-tier gauges show what the backend reports (#25). Capacities come
from the local backend'sMemoryInfo— the device's total for a GPU
backend, host RAM for the CPU one — instead of the fixed 8/4/2 GiB
constants the gauges used to be drawn against, and a tier the engine has no
number for is drawn without a ratio rather than against a plausible one. -
The dashboard says the spill manager is library-only (#25).
SpillManageris not on the query path: no operator registers a batch with
it, so the tier gauges and spill counters stay at zero however much memory
a query used. The Telemetry panel says so (spill library only: no query registers batches), as do the README,STATUS.mdand
docs/architecture.md. Wiring it in is not a matter of calling it from the
join — the build side is probed by every batch — and waits on the streaming
aggregate (#29). -
OperatorSnapshotandTelemetrySnapshotgained public fields
(fallback_batches,capacity). Both are exhaustively constructible, so a
downstream struct literal over them no longer compiles. They are snapshot
types read field by field in practice, which is why they stay plain structs
rather than becoming#[non_exhaustive]— a reader should be able to
destructure one. -
An unavailable backend names the backend, not the way it was asked for.
oxide-worker --backend cudareaches the same code asOXIDE_BACKEND, so
the message no longer sends the reader to a variable they did not set. -
Logs are uncoloured when stderr is not a terminal. ANSI escapes in a
redirected log file are noise, and they break a grep forfield=value.
Fixed
oxide sqlprints the columns of an empty result. The CSV header and
the JSON[]are written from the plan's schema, which an empty result
still has.