Skip to content

Releases

NVIDIA Recovery v1.0.1

Choose a tag to compare

@github-actions github-actions released this 27 Oct 20:16
v1.0.1
8cb8739

NVIDIA Recovery v1.0.1

This patch release follows v1.0.0. It completes durable recovery tracing in
the Slurm Advisor Controller, fixes OTLP/HTTP export from the Python services,
and moves Attribution evidence ingestion back to the provider's V2 API.

Highlights

  • The Slurm Advisor Controller continues the recovery trace that Job Manager
    starts on each Advice. Reconcile and scheduler-action spans join the backend
    trace, and the controller reports scheduler-command milestones that Job
    Manager uses for restart completion and latency. Milestones are delivered
    by a bounded background worker and do not block reconciliation.
  • Python services now send OTLP/HTTP traces and logs to the /v1/traces and
    /v1/logs paths of the shared collector endpoint instead of the bare base
    URL, which collectors reject. http/protobuf is now accepted as an
    OTEL_EXPORTER_OTLP_PROTOCOL value. Previously it stopped trace setup at
    startup and disabled OTLP logging.
  • The Attribution adapter sends log and metric observations through the
    provider's V2 observation API, consistent with sessions and results. Loki
    streams and Prometheus samples are converted to the V2 entry_timeseries
    form, and metric queries use the job's query window instead of a fixed
    "last 10 minutes" interval.

Compatibility and upgrade notes

  • No Gateway HTTP/gRPC or Attribution Adapter contract changes. Milestones use
    the existing gateway connection and credentials.
  • New optional slurm-advisor-controller.controller.milestoneReporting Helm
    values (queueCapacity: 256, rpcTimeout: 2s, maxAttempts: 3,
    retryDelay: 100ms, shutdownTimeout: 2s), exposed to the controller as
    SLURM_ADVISOR_MILESTONE_* environment variables. All five values must be
    positive. The chart schema rejects zero for queueCapacity and
    maxAttempts; a zero duration renders but stops the controller at startup.
    Existing values files need no changes. See the
    Slurm Advisor Controller chart guide
    for sizing the queue for a backend outage.
  • When only the shared OTEL_EXPORTER_OTLP_ENDPOINT is set, controller traces
    now use the shared protocol (default grpc) instead of defaulting to HTTP.
    Only a trace-specific endpoint keeps the OTLP http/protobuf default. Set
    global.otelcollector.endpoint to the collector base URL without a signal
    path. See the Helm guide.
  • With no trace exporter configured, services no longer sample traces. Context
    still propagates, marked unsampled. A controller trace exporter that cannot be
    created is logged and disabled instead of stopping the controller.
  • Audit events and stored anomalies now record trace and span IDs only for
    sampled spans, so trace links never point at traces that were not exported.
    Controller audit payloads add generation, recovery_sequence_id,
    trace_id, and span_id. Consumers that expect IDs on every event should
    treat them as optional.
  • Attribution observations are filtered by the window the adapter declares,
    not by the provider's V1 job-timing rules. This can change which evidence
    feeds node-attribution and node-exclusion decisions.
  • Operators build service images from this source revision and configure the
    chart with their own registry references. This release does not publish
    product container images. Go builds now require Go 1.27.3.

Known limitations

  • Recovery milestones are held only in controller memory. A controller restart
    can leave a recovery sequence without a recorded completion.
  • Advice Operator source and its standalone chart are restored in the
    repository for downstream consumers but remain excluded from release
    archives and are not a supported release path.

Qualification boundaries

  • Local end-to-end runs now deploy an OTLP collector and Tempo and require
    each restart and DAG-release recovery to form one trace across the backend
    and controller, with Job Manager and controller logs exported through the
    shared endpoint. This does not establish live scheduler, GPU, or production
    collector qualification.
  • Packaging validates reproducible source and Helm archives and runs Helm
    lint/render. These checks do not establish installation or recovery
    qualification.
  • Private Attribution qualification is separate and is not a public release
    prerequisite. Dependency vulnerability and license review remains part of
    maintainer release approval.
  • Archive checksums detect byte changes; they do not establish signing,
    provenance, or secret scanning.

See the release procedure,
installation guide, and
support policy for release and operational boundaries.