[P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated

[aditya-shantanu]

Dedup note: verified against all open+closed issues and PRs 2026-07-30 —
no prior report. Adjacent items ruled out: PR #558/#577 fixed the
benchmark's missing credentials with mTLS pod certs (per-handshake
reload) and never touch the ateclient bearer-token path; PR #248 /
issue #222 introduced this token code without noting the no-refresh
limitation; #459/PR #537 cover credbundle cert rotation (mTLS side only).

Summary

internal/ateclient requests one ServiceAccount token with
ExpirationSeconds: 3600 at dial time and wraps it in a static
per-RPC credential. The token is never re-minted. Any client process that
lives longer than one hour starts failing every RPC with
Unauthenticated: invalid bearer token: jwt has expired and never recovers
without a full reconnect.

Evidence

• internal/ateclient/builder.go:232-247 — bearerTokenDialOption mints the
  token once; bearerTokenCreds is a string type whose
  GetRequestMetadata (builder.go:252) returns the same literal token for
  the lifetime of the connection.
• Empirically confirmed by the 2026-07-30 overnight soak
  (stability/overnight-test.yaml, 8h run on GKE substrate-poc):
  • Job started 05:42Z; first jwt has expired error at 06:49:18Z (~1h07m in).
  • From then until the run ended, every gRPC call failed:
    176,707 Unauthenticated errors across ResumeActor / SuspendActor /
    CycleResume, and all 8 DeleteActor cleanup calls failed, leaking the
    soak actors.

Impact

• The reference client library (used by kubectl-ate, stability/stresstest,
  and as the model for external clients) cannot run a workload for more than
  1 hour.
• Cleanup paths that run at process end (e.g. stresstest's actor deletion) are
  guaranteed to fail in long runs, leaking actors and workers.
• Anyone copying this library pattern into a production client ships the
  same time bomb.

Reproduction

1. ateclient.NewClient(...) against any cluster.
2. Sleep 65 minutes.
3. Any RPC → Unauthenticated: invalid bearer token: jwt has expired.

Suggested fix

Implement a refreshing credentials.PerRPCCredentials: cache the token and
its expiry, re-mint via the TokenRequest API when within a skew window
(e.g. 5 min) of expiry, under a mutex. GetRequestMetadata is already called
per-RPC, so refresh can be lazy and transparent.

[EItanya]

This definitely needs to be fixed, but personally I would rather focus on what we want our user auth strategy to be in Substrate and then build that out, rather than fixing bugs with the current implementation.