feat(gantry): improve cold-start distribution and benchmark automation

[Vaibhav Patel (vpatelsj)]

Summary

This PR expands the Gantry benchmark into a reproducible, observable large-cluster workflow and includes targeted Gantry product changes identified while exercising that workflow. Product behavior changes are listed first; benchmark-only infrastructure and tuning are separated below.

Product changes

Gantry cold-start and transfer behavior

• Activate speculative manifest prefetch for the production live stream-through path. The prefetcher waits for the manifest to appear in containerd, records its layers, and uses the manifest digest as the stable coordinator-election key.
• Limit remote speculative prefetch to three deterministic manifest consumers, cap remote dispatch at 64 concurrent groups, and spread dispatch with up to one second of deterministic jitter. Self-selected local seed pulls still run independently.
• Increase the complete peer-fetch safety ceiling from 60 seconds to 15 minutes. The existing HTTP/2 read-idle detector continues to detect connections with no read progress after 10 seconds.
• Copy known-length mirror and peer-transfer bodies through pooled 1 MiB buffers and advertise a 1 MiB HTTP/2 receive frame size.
• Enforce exact declared body lengths for peer transfers. Range requests now finish positioning before returning 206 Partial Content; positioning failures return 500.
• Populate libp2p peerstore addresses from the Kubernetes membership view before coordination RPCs dial a peer.
• Deduplicate bootstrap peers by peer ID and use bounded bootstrap dialing: batches of 8, a cumulative quorum of 4 successful connections, and at most 32 attempts.
• Stop containerd Has and Open retry paths immediately when their context has already been canceled.

Product configuration and observability

• Set image_pull_progress_timeout = "15m" in the containerd configuration generated for Unbounded-managed nodes. Production containerd debug logging is not enabled; debug logging remains confined to the benchmark-only containerd manifest.
• Add optional Go pprof endpoints. Profiling is disabled by default, accepts only loopback listeners, and does not own Gantry data-plane availability.
• Add metrics for peer busy/stall timestamps, completed mirror responses, per-layer completion timestamps, prefetch group outcomes, and stream-completion-to-containerd-inventory observation times.
• Initialize known peer-fetch and DHT outcome label sets at zero so the series exist before the first event.
• Change p2p_prefetch_pullers_per_manifest from linear buckets 1..10 to exponential buckets 1..1024.
• Bound per-layer metric labels to the current manifest and remove the previous manifest's label values.
• Exclude environment files, kubeconfigs, PEM/key files, SSH keys, and cloud CLI configuration from repository-root container build contexts through .dockerignore.

Benchmark workflow

• Add a standalone live monitor that displays benchmark pod states, per-node layer download/unpack progress, peer outcomes, bytes served, and node CPU/memory utilization.
• Add streaming command execution and periodic Kubernetes Job progress reporting for long-running benchmark phases.
• Add reusable Gantry image pools with atomic ready/claim metadata, pool status reporting, and support for Gantry-only runs against a retained baseline.
• Add fresh-image, adopted-image, and pooled-image preparation paths using digest-pinned references and payload SHA-256 validation.
• Preserve partial CPU profiles and add a wrapper for starting a fresh full benchmark lifecycle.
• Capture phase-bounded Gantry metrics, per-pod peer-byte counters, DHT/peer-fetch diagnostics, and stream-to-containerd commit observations.
• Capture raw Prometheus range data for node, disk, network, containerd, gRPC, Gantry, prefetch, and layer-completion metrics.
• Capture structured containerd journal events for each benchmark phase.
• Strengthen preflight and result completeness checks for monitoring coverage, Gantry pod/revision coverage, baseline routing isolation, peer activity, origin bytes, peer bytes, Azure telemetry, and fallback pulls.
• Report image-pool and run cleanup status in retained artifacts.

Benchmark infrastructure and operations

• Add an Azure deployment workflow for a dedicated benchmark environment containing AKS, separate baseline and Gantry ACRs, private endpoints and DNS, an operator VM, Log Analytics, and Prometheus monitoring.
• Add operator VM bootstrap, image build, image-pool, run, status, and watch workflows using managed identity and systemd services.
• Add a source-carrier image used to transfer the repository checkout to the operator VM.
• Add a benchmark-only privileged containerd configuration DaemonSet. It enables debug logging, sets a 15-minute pull-progress timeout, configures six concurrent downloads, restarts containerd, and verifies the effective configuration.
• Expand benchmark monitoring with node-exporter, containerd metrics, Gantry PodMonitor rules, and structured containerd journal collection.
• Add a profiling workflow that selects Gantry pods by measured CPU usage and captures CPU, heap, and goroutine profiles.
• Restore benchmark routing and resources during cleanup while retaining run artifacts and status.

Documentation

• Expand the benchmark README and runbook for deployment, operator workflows, image pools, monitoring, profiling, cleanup, and artifact interpretation.
• Add pull-latency analysis and update retained benchmark results with the new telemetry fields.
• Add an example deployment environment file and ignore local deployment configuration.

Test coverage

• Add or expand Gantry tests for prefetch coordinator election, concurrency and jitter, manifest prefetch under live stream-through, peer address resolution, bootstrap dialing, exact-length stream copying, range errors, cancellation, profiling, per-layer metrics, and containerd commit timing.
• Add benchmark tests for streaming commands, image pools and adoption, Job progress, performance telemetry, peer telemetry, monitoring preflight, and direct-mode behavior.
• Add monitor tests for pod classification, node grids, resource summaries, and Prometheus response handling.
• Validate the production node-start containerd configuration independently from the benchmark-only containerd manifest.

[Copilot]

Pull request overview

This PR expands Gantry agent and gantry-benchmark observability by adding new Prometheus metrics and benchmark-side telemetry capture, while also hardening prefetch/peer addressing behaviors and improving benchmark deployment/run workflows.

Changes:

• Add new Gantry metrics (mirror response completion timestamps, peer fetch timestamps/latencies, containerd commit observation timing) and wire them through agent/mirror/commit tracking paths.
• Improve cold-start prefetch reliability and dispatch behavior (live stream-through manifest prefetch fix, bounded group concurrency, coordinator election, jitter/rotation).
• Enhance gantry-benchmark operational tooling (full-stack deploy workflow, node observer DaemonSet + PodMonitors, richer operator VM status/watch output, live monitor tool).

Reviewed changes

Copilot reviewed 68 out of 69 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/agent/phases/nodestart/cri_test.go	Adds coverage asserting generated containerd config includes pull-progress timeout and debug stanza.
pkg/agent/phases/nodestart/assets/containerd.toml	Updates default containerd config template to enable debug logging and set image pull progress timeout.
internal/gantry/mirror/mirror.go	Adds hook for mirror response completion and ensures manifest prefetch fires for peer-served manifests in live stream-through mode.
internal/gantry/mirror/mirror_prefetch_test.go	Adds regression test ensuring prefetch triggers for peer-served manifests when live stream-through is enabled.
internal/gantry/mirror/byte_metrics_test.go	Extends mirror byte-metric tests to assert response-completion hook behavior across cache/peer/origin paths.
internal/gantry/discovery/discovery.go	Deduplicates/merges peer AddrInfo entries and implements bounded bootstrap dialing with accumulated success quorum.
internal/gantry/discovery/discovery_test.go	Adds unit test for merging AddrInfo by peer ID and combining addresses.
internal/gantry/containerdstore/store.go	Preserves context cancellation/deadline errors rather than mapping them to availability errors.
internal/gantry/containerdstore/store_test.go	Adds tests ensuring context errors are not remapped to ErrUnavailable.
internal/gantry/config/config.go	Introduces prefetch dispatch config knobs and increases default peer fetch timeout to 15m with updated rationale.
internal/gantry/config/config_test.go	Updates default expectations and adds env/flag/validation tests for new prefetch dispatch settings.
internal/gantry/coldstart/prefetch.go	Implements coordinator election, deterministic dispatch jitter/rotation, bounded remote group concurrency, and per-group outcome metrics.
internal/gantry/coldstart/prefetch_test.go	Adds tests for group outcome reporting, bounded concurrency, and deterministic coordinator dispatch behavior.
internal/gantry/coldstart/prefetch_internal_test.go	Adds internal tests for deterministic dispatch planning and disabled jitter behavior.
internal/gantry/coldstart/coldstart.go	Extends options and metrics hooks for prefetch group tracking and defaults max concurrent groups.
internal/gantry/coldstart/coldstart_test.go	Updates helper to pass MetricsHooks into resolver construction.
images/gantry-benchmark-source/Containerfile	Adds a source-carrier image for private benchmark branches/revisions.
hack/gantry-benchmark/RUNBOOK.md	Updates runbook guidance for deploy.sh-first workflow, 1000-node contract defaults, and containerd config application steps.
hack/gantry-benchmark/RESULTS.md	Adds new benchmark result rows and expanded interpretation/footnotes for recent runs.
hack/gantry-benchmark/README.md	Documents full-stack deploy workflow, required instrumentation, and new live monitor usage/details.
hack/gantry-benchmark/operator-vm-watch.sh	Adds run-command locking, uses local status script via Fine Free File Command (@file), and prints timestamped snapshot headers.
hack/gantry-benchmark/operator-vm-status.sh	Expands operator status reporting for image build/push state, job progress rendering, and active operations.
hack/gantry-benchmark/operator-vm-run.sh	Adds structured logging, supports adopt-prepared images, and improves lifecycle messaging.
hack/gantry-benchmark/operator-vm-provision.sh	Plumbs source image/revision and adopt-image parameters into operator VM bootstrap/run invocation.
hack/gantry-benchmark/operator-vm-build-images.sh	New script to build/push deployment images on the operator VM and persist digest state.
hack/gantry-benchmark/manifests/monitoring.yaml.tmpl	Expands metric allowlist, adds gantry_benchmark labeling, and introduces node-observer DaemonSet + PodMonitor.
hack/gantry-benchmark/manifests/containerd.yaml	Adds a DaemonSet to enforce benchmark containerd drop-in config (debug, pull timeout, download concurrency) with readiness checks and restart gating.
hack/gantry-benchmark/Makefile	Adds deploy/monitor targets, operator script checks, and prepare-adopt target for direct mode.
hack/gantry-benchmark/deploy.env.example	Adds full-stack deploy contract example with adoption knobs and tightened benchmark gate defaults.
hack/gantry-benchmark/.gitignore	Ignores deploy.env alongside env.local.
hack/cmd/gantry-benchmark/run.go	Captures and persists per-phase diagnostic deltas and performance telemetry artifacts for baseline and Gantry phases.
hack/cmd/gantry-benchmark/results.go	Extends JSON results schema with diagnostic measurement and performance telemetry artifact references.
hack/cmd/gantry-benchmark/preflight.go	Adds Prometheus scrape coverage checks and supports multiple Azure diagnostic settings response shapes.
hack/cmd/gantry-benchmark/preflight_monitoring_test.go	Adds unit tests for metric coverage retry behavior and timeout error formatting.
hack/cmd/gantry-benchmark/performance_telemetry_test.go	Adds tests for query_range generation, bounded containerd query selection, and response/pod coverage validation.
hack/cmd/gantry-benchmark/peer_telemetry.go	Adds Gantry diagnostic snapshot/timestamp capture, per-pod delta computation, and validation helpers.
hack/cmd/gantry-benchmark/peer_telemetry_test.go	Adds tests for timestamp windowing, required final-layer timestamps, metric keying, and delta subtraction behavior.
hack/cmd/gantry-benchmark/main.go	Adds timestamped stdout/stderr writers, and introduces prepare-adopt CLI subcommand.
hack/cmd/gantry-benchmark/job.go	Adds live pull job progress reporting, and persists per-pod start/finish timing details in results.
hack/cmd/gantry-benchmark/job_test.go	Extends job observation test to validate per-pod timing capture.
hack/cmd/gantry-benchmark/job_progress_test.go	Adds tests for progress state classification and progress reporter lifecycle behavior.
hack/cmd/gantry-benchmark/image.go	Streams long-running build/push output with prefixing, adds payload generation/hashing progress, and adds prepare-adopt implementation.
hack/cmd/gantry-benchmark/image_test.go	Extends tests to assert progress output and validate adopt-prepared image behavior and input validation.
hack/cmd/gantry-benchmark/gantry_only.go	Adds diagnostic and performance telemetry capture to Gantry-only benchmark mode results.
hack/cmd/gantry-benchmark/enable.go	Passes node OS/arch to monitoring manifest and waits for node-observer rollout.
hack/cmd/gantry-benchmark/enable_test.go	Validates expanded monitoring manifest contents and asserts containerd benchmark manifest expectations.
hack/cmd/gantry-benchmark/config.go	Adds BENCHMARK_JOB_PROGRESS_INTERVAL config field and default.
hack/cmd/gantry-benchmark/command.go	Adds optional streaming command runner, prefix writer, timestamp writer, and makes writeAll nil-safe.
hack/cmd/gantry-benchmark/command_streaming_test.go	Adds tests for streaming delivery, stderr capture, prefix/timestamp writer behaviors and concurrency safety.
hack/cmd/gantry-benchmark/azure_preflight_test.go	Adds tests for decoding Azure diagnostic settings in both CLI and ARM response shapes.
hack/cmd/gantry-benchmark-monitor/pods.go	Implements Kubernetes watch-based pod state tracking for live monitoring.
hack/cmd/gantry-benchmark-monitor/pods_test.go	Adds tests for pod state classification and snapshot aggregation.
hack/cmd/gantry-benchmark-monitor/monitor_test.go	Adds tests for Prometheus range parsing/aggregation and monitor rendering.
deploy/gantry/configmap.yaml.tmpl	Updates Gantry default config to include new prefetch dispatch settings and updated peer fetch timeout.
cmd/gantry/stream_commit_tracker.go	Tracks per-stream completion timestamps and reports observation durations in completion-time order.
cmd/gantry/stream_commit_tracker_test.go	Adds tests for duration reporting and ordering guarantees.
cmd/gantry/prefetch_manifest_test.go	Adds tests for manifest open retry loop under live stream-through timing.
cmd/gantry/membership_peer_resolver_test.go	Adds tests ensuring membership-based resolver installs pod addresses and rejects identity mismatches.
cmd/gantry/main.go	Wires new metrics hooks, adds peerstore-aware membership resolver, and implements manifest open retry + PrefetchManifestChildren usage.
cmd/gantry/agent_byte_metrics_test.go	Updates startup metric materialization expectations to include new metric families.
.dockerignore	Expands exclusions for env files, kubeconfigs, keys, and cloud credential directories.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.