Repository navigation
Releases
Releases · dsx-ai-factory/NVIDIA-Recovery
NVIDIA Recovery v1.0.1
NVIDIA Recovery v1.0.1
This patch release follows v1.0.0. It completes durable recovery tracing in
the Slurm Advisor Controller, fixes OTLP/HTTP export from the Python services,
and moves Attribution evidence ingestion back to the provider's V2 API.
Highlights
- The Slurm Advisor Controller continues the recovery trace that Job Manager
starts on each Advice. Reconcile and scheduler-action spans join the backend
trace, and the controller reports scheduler-command milestones that Job
Manager uses for restart completion and latency. Milestones are delivered
by a bounded background worker and do not block reconciliation. - Python services now send OTLP/HTTP traces and logs to the
/v1/tracesand
/v1/logspaths of the shared collector endpoint instead of the bare base
URL, which collectors reject.http/protobufis now accepted as an
OTEL_EXPORTER_OTLP_PROTOCOLvalue. Previously it stopped trace setup at
startup and disabled OTLP logging. - The Attribution adapter sends log and metric observations through the
provider's V2 observation API, consistent with sessions and results. Loki
streams and Prometheus samples are converted to the V2entry_timeseries
form, and metric queries use the job's query window instead of a fixed
"last 10 minutes" interval.
Compatibility and upgrade notes
- No Gateway HTTP/gRPC or Attribution Adapter contract changes. Milestones use
the existing gateway connection and credentials. - New optional
slurm-advisor-controller.controller.milestoneReportingHelm
values (queueCapacity: 256,rpcTimeout: 2s,maxAttempts: 3,
retryDelay: 100ms,shutdownTimeout: 2s), exposed to the controller as
SLURM_ADVISOR_MILESTONE_*environment variables. All five values must be
positive. The chart schema rejects zero forqueueCapacityand
maxAttempts; a zero duration renders but stops the controller at startup.
Existing values files need no changes. See the
Slurm Advisor Controller chart guide
for sizing the queue for a backend outage. - When only the shared
OTEL_EXPORTER_OTLP_ENDPOINTis set, controller traces
now use the shared protocol (defaultgrpc) instead of defaulting to HTTP.
Only a trace-specific endpoint keeps the OTLPhttp/protobufdefault. Set
global.otelcollector.endpointto the collector base URL without a signal
path. See the Helm guide. - With no trace exporter configured, services no longer sample traces. Context
still propagates, marked unsampled. A controller trace exporter that cannot be
created is logged and disabled instead of stopping the controller. - Audit events and stored anomalies now record trace and span IDs only for
sampled spans, so trace links never point at traces that were not exported.
Controller audit payloads addgeneration,recovery_sequence_id,
trace_id, andspan_id. Consumers that expect IDs on every event should
treat them as optional. - Attribution observations are filtered by the window the adapter declares,
not by the provider's V1 job-timing rules. This can change which evidence
feeds node-attribution and node-exclusion decisions. - Operators build service images from this source revision and configure the
chart with their own registry references. This release does not publish
product container images. Go builds now require Go 1.27.3.
Known limitations
- Recovery milestones are held only in controller memory. A controller restart
can leave a recovery sequence without a recorded completion. - Advice Operator source and its standalone chart are restored in the
repository for downstream consumers but remain excluded from release
archives and are not a supported release path.
Qualification boundaries
- Local end-to-end runs now deploy an OTLP collector and Tempo and require
each restart and DAG-release recovery to form one trace across the backend
and controller, with Job Manager and controller logs exported through the
shared endpoint. This does not establish live scheduler, GPU, or production
collector qualification. - Packaging validates reproducible source and Helm archives and runs Helm
lint/render. These checks do not establish installation or recovery
qualification. - Private Attribution qualification is separate and is not a public release
prerequisite. Dependency vulnerability and license review remains part of
maintainer release approval. - Archive checksums detect byte changes; they do not establish signing,
provenance, or secret scanning.
See the release procedure,
installation guide, and
support policy for release and operational boundaries.