Operational scripts are token-unaware and break under --auth-mode=required

[balajinvda]

Follow-up to #555. With the agent running --auth-mode=required, scripts that call its HTTP API without an Authorization header get a bare unauthorized. Reproduced on nvcf-dgxc-k8s-aws-usw2-dev2:

Creating checkpoint for vllm-small/vllm...
Error: Failed to get checkpoint ID from response
Response: unauthorized
[ERROR] FAILED at: Checkpoint

scripts/checkpoint.sh is fixed on the #555 branch. This issue covers the rest.

Actual scope

16 scripts mention port 8081, but most are comments, echoed help text, or delegate to checkpoint.sh. Only three make a real authenticated call:

Script	Line	Call
test-e2e.sh	803	POST /v1/restore
test-agent-driven-e2e.sh	223	POST /v1/restore
bench-cross-node-restore.sh	84	GET /debug/pprof/profile (via kubectl exec)

deploy-agent.sh:115 hits /health, which is unauthenticated by design and needs no change. deploy-agent.sh:124, deploy-all.sh:70, deploy-k8s.sh:79 print instructions rather than calling. nccl-trace-quick.sh:22 is a usage comment showing an unauthenticated POST /v1/checkpoint and should be updated so the documented example is not misleading.

pprof is deliberately gated: it is registered on the guarded router and is not in unauthenticatedPaths, per the comment in internal/agent/auth.go -- "profiles leak memory contents and goroutine state".

Fix direction

Add scripts/lib/agent-auth.sh with one token lookup, and source it from every script that calls the API, replacing the inline helper currently in checkpoint.sh. An absent Secret must yield no header rather than an error, since the same scripts have to work against a cluster installed without agent.auth.enabled.

For calls made inside the agent pod via kubectl exec (bench-cross-node-restore.sh), use the NVSNAP_AGENT_TOKEN env var the chart already injects into the container instead of piping the token in from outside.

Why it matters

The defaults (auth.enabled=false, mode=permissive) are unaffected, so this is invisible until someone turns on the mode the feature exists to provide. Anyone who does gets a broken checkpoint/restore toolchain and an error message that does not say which credential is missing.

NVCA's calls to the agent API should be audited the same way before required is recommended.

[balajinvda]

NVCA audit done. No change needed on the NVCA side for --auth-mode=required.

NVCA never calls the agent API. pkg/nvca/nvsnap/client.go (added by #208) targets nvsnap-server only:

const DefaultBaseURL = "http://nvsnap-server.nvsnap-system.svc.cluster.local:8080"

CreateCheckpoint, GetCheckpoint, LookupCheckpoints, GetPVCPromoteState and DeleteCheckpoint all route through do() to /api/v1/checkpoints* on :8080. Checked every Go file #208 adds or modifies for :8081 or nvsnap-agent -- no hits. nvca on main contains no nvsnap code at all, so #208 is the entire surface.

So the remaining item on this issue is closed: agent auth does not break NVCA, because NVCA is not an agent client.

The audit did surface a larger gap in the component NVCA does depend on: nvsnap-server has no authentication and the chart defaults its Service to type: LoadBalancer. Filed separately as #735, since it is a different component and a different fix. Note the ordering constraint there -- adding auth to nvsnap-server would break NVCA exactly the way required broke these scripts, so the client needs the token first.