Skip to content

Affected >= oci-java-sdk-common:2.91.0 (not 3.xx.x): "inFlightRefresh" is stuck when Leader is exit before "try/catch" block #777

Description

@k2chogori

Affected >= oci-java-sdk-common:2.91.0 (not 3.xx.x): "inFlightRefresh" is stuck when Leader is exit before "try/catch" block

X509FederationClient leaves inFlightRefresh incomplete when the refresh leader fails before cleanup

Description

Summary

X509FederationClient can become permanently stuck after an exception during session-key or
certificate refresh.

The single-flight implementation assigns a new CompletableFuture to inFlightRefresh, but several
operations that can throw are executed before the try/catch/finally responsible for completing and
clearing that future.

If one of those operations throws:

  1. The Leader exits without completing inFlightRefresh.
  2. inFlightRefresh remains non-null and incomplete.
  3. Every subsequent caller becomes a Follower.
  4. Each Follower waits one minute and fails with Timed out waiting for security token refresh.
  5. The affected X509FederationClient does not recover. Recreating the authentication provider or
    restarting the process is required.

Affected versions

Confirmed by a standalone reproducer:

  • oci-java-sdk-common:2.91.0
  • oci-java-sdk-common:2.93.0

Production stack traces from both observed incidents reported Oracle-JavaSDK/2.92.0.

The issue was introduced by commit
6a737779038869e1ca2948868a225736befdae0e,
which released version 2.91.0 and added the single-flight implementation.

The relevant file is unchanged between public tags v2.91.0 and v2.93.0.

This can be verified with:

git diff v2.91.0 v2.93.0 -- \
  bmc-common/src/main/java/com/oracle/bmc/auth/internal/X509FederationClient.java

The command produces no output.

Source-code evidence

In
X509FederationClient.refreshAndGetSecurityTokenInner,
the Leader publishes the incomplete future:

future = new CompletableFuture<>();
inFlightRefresh = future;
iAmTheLeader = true;

The following operations are then executed before the cleanup try/finally:

sessionKeySupplier.refreshKeys();
((Refreshable) leafCertificateSupplier).refresh();
AuthUtils.getTenantIdFromCertificate(...);
((Refreshable) supplier).refresh();

The cleanup block starts only later:

try {
    securityTokenAdapter = getSecurityTokenFromServer();
    String token = securityTokenAdapter.getSecurityToken();
    future.complete(token);
    return token;
} catch (Exception e) {
    future.completeExceptionally(e);
    throw new BmcException(false, "Error refreshing security token.", e, null);
} finally {
    inFlightRefresh = null;
}

An exception thrown before this try bypasses both:

future.completeExceptionally(e);
inFlightRefresh = null;

The leaf-certificate block catches only RefreshFailedException. In the production failures described
below, URLBasedX509CertificateSupplier exhausted its retries and threw:

java.lang.IllegalArgumentException: Open stream of certificate failed.
Caused by: java.io.FileNotFoundException:
http://169.254.169.254/opc/v2/identity/cert.pem

This exception escaped at:

URLBasedX509CertificateSupplier.refresh
X509FederationClient.refreshAndGetSecurityTokenInner:248

Before the introducing commit, refresh was performed entirely under the existing synchronized block
and there was no persistent inFlightRefresh state that could be left incomplete.

Minimal reproduction

Use this dependency:

<dependency>
  <groupId>com.oracle.oci.sdk</groupId>
  <artifactId>oci-java-sdk-common</artifactId>
  <version>2.93.0</version>
</dependency>
  1. Construct an X509FederationClient with an X509CertificateSupplier that also implements
    javax.security.auth.Refreshable.
  2. Make its refresh() method throw IllegalArgumentException, matching the terminal exception from
    URLBasedX509CertificateSupplier.
  3. Call refreshAndGetSecurityToken(). This caller becomes the Leader and exits with the simulated
    certificate failure.
  4. Inspect inFlightRefresh: it is non-null and isDone() is false.
  5. Call refreshAndGetSecurityToken() from another thread. It becomes a Follower and times out after
    one minute.

Observed output using the real oci-java-sdk-common:2.93.0 artifact:

Loaded SDK artifact: .../oci-java-sdk-common/2.93.0/oci-java-sdk-common-2.93.0.jar

[EXPECTED FAILURE]
  Leader stopped with: Simulated failure reading IMDS cert.pem

[SDK STATE]
  Does inFlightRefresh still exist? YES
  Has inFlightRefresh completed?       NO

[FOLLOWER RESULT]
  Exception: com.oracle.bmc.model.BmcException
  Message:   (-1, null, false) Timed out waiting for security token refresh.

VERDICT: BUG REPRODUCED

The reproducer does not require OCI credentials or network access.

Production evidence

The same sequence was observed independently in two long-running Java processes on OCI compute
instances:

[Leader] About to refresh security token.
Refreshing session keys.
Attempt 1 to open stream of certificate failed.
Attempt 2 to open stream of certificate failed.
Attempt 3 to open stream of certificate failed.
java.lang.IllegalArgumentException: Open stream of certificate failed.
Caused by: java.io.FileNotFoundException:
http://169.254.169.254/opc/v2/identity/cert.pem

After the Leader terminated, subsequent callers repeatedly logged:

[Follower] Waiting on leader's result.
com.oracle.bmc.model.BmcException:
Timed out waiting for security token refresh.
Caused by: java.util.concurrent.TimeoutException

In one occurrence, the logs contained 62 Follower timeouts. Authenticated Monitoring operations
continued failing until the process restarted. Restarting created a new authentication provider and
immediately restored token refresh.

There is an additional timeout mismatch: the certificate supplier's three retries can take longer
than the hard-coded one-minute Follower wait, so Followers may time out even while a slow Leader is
still retrying. The persistent failure described in this issue remains after that Leader has exited
because its future is never completed or cleared.

Expected behavior

Every Leader exit path should:

  1. Complete the shared future, normally or exceptionally.
  2. Clear inFlightRefresh.
  3. Allow a later caller to become a new Leader and retry.

Followers should receive the Leader's original exception rather than wait for a future that can never
complete.

Suggested fix

Move all Leader operations under the same try/catch/finally:

if (iAmTheLeader) {
    try {
        if (refreshKeys) {
            sessionKeySupplier.refreshKeys();
        }

        refreshLeafCertificate();
        validateTenancy();
        refreshIntermediateCertificates();

        securityTokenAdapter = getSecurityTokenFromServer();
        String token = securityTokenAdapter.getSecurityToken();
        future.complete(token);
        return token;
    } catch (Exception e) {
        future.completeExceptionally(e);
        throw new BmcException(false, "Error refreshing security token.", e, null);
    } finally {
        inFlightRefresh = null;
    }
}

(content is generated by AI, but was read by human and it makes sense)

Metadata

Metadata

Assignees

No one assigned

    Labels

    SDKIssue pertains to the SDK itself and not specific to any service

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions