[SPARK-38954][CORE][WIP] Cloud Credential Refresh and Distribution Without Kerberos

[parthchandra]

What changes were proposed in this pull request?

This PR enables Spark's existing credential distribution mechanism (driver obtains tokens, pushes them to executors, refreshes before expiry) to work without Kerberos. Today this mechanism is unconditionally gated
on Kerberos being enabled. This change adds a single configuration switch (spark.security.credentials.directProviders.enabled) that lifts the Kerberos gate on token collection.

Changes:

• HadoopDelegationTokenManager.renewalEnabled now also returns true when the new config is enabled.
• obtainDelegationTokens(creds) and obtainTokensAndScheduleRenewal() branch on whether Kerberos credentials are present: if yes, call providers inside doLogin()/doAs() as before; if not (but the switch is
  on), call providers directly. Providers that require Kerberos self-gate via delegationTokensRequired returning false.
• Per-provider try/catch for failure isolation — one provider throwing does not block others.
• SupportsDelegationToken gains an overridable tokenManagerRequired() hook so scheduler backends can activate the token manager based on the new config.
• Hard startup failure if spark.network.crypto.enabled is not true when the switch is active (prevents bearer tokens over plaintext RPC).
• WARN log when the config is set but no providers are discovered via ServiceLoader.

Design document: SPIP: Cloud Credential Refresh and Distribution Without Kerberos

Why are the changes needed?

In non-Kerberos deployments (cloud, internal IdP, cross-account access on YARN), each executor independently authenticates against identity services, causing thundering-herd load amplification, inconsistent
credential state, and no centralized control over credential rotation. The existing distribution infrastructure is already provider-agnostic (Kafka proves this) — only the activation gates prevent it from running
without Kerberos.

Does this PR introduce any user-facing change?

Yes. New configuration:

Config	Default	Description
spark.security.credentials.directProviders.enabled	false	Enables delegation token collection and renewal without Kerberos. When true, the manager starts even if Hadoop security is not enabled, calling all
providers whose delegationTokensRequired returns true. Requires spark.network.crypto.enabled=true.

Existing deployments are unaffected (config defaults to false). When Kerberos IS present, all code paths are unchanged.

How was this patch tested?

New unit test suite NonKerberosCredentialsSuite (7 tests):

• renewalEnabled returns true/false based on config
• Providers are called without Kerberos when config is enabled
• Providers with delegationTokensRequired=false are skipped
• A failing provider does not prevent other providers from running
• Individual provider can be disabled via spark.security.credentials.<service>.enabled
• Startup fails if spark.network.crypto.enabled is not true

Existing HadoopDelegationTokenManagerSuite (4 tests) passes unmodified — verifies no regression in Kerberos deployments.

Was this patch authored or co-authored using generative AI tooling?

Yes, Co-authored by Claude Code (opus 4.6)

[dongjoon-hyun]

Thank you, @parthchandra .