Skip to content

Fix: Reuse shared OPA HTTP client#31

Merged
NickLarsenNZ merged 1 commit into
stackabletech:mainfrom
Saul-STFC:fix/thread-fix
Jul 2, 2026
Merged

Fix: Reuse shared OPA HTTP client#31
NickLarsenNZ merged 1 commit into
stackabletech:mainfrom
Saul-STFC:fix/thread-fix

Conversation

@Saul-STFC

Copy link
Copy Markdown
Contributor

This change replaces the OPA SDK's per-request HTTP transport with a shared java.net.http.HttpClient Addressing #30

The majority of this PR is based on the code provided by:
#30 (comment) - Thanks!! :)

Background

The existing SDK transport can create a new HttpClient for each authorisation request. Under NiFi load, this may create large numbers of selector and executor threads, eventually causing native thread exhaustion and errors such as:

The new PooledHttpClient reuses one HTTP client, connection pool, selector infrastructure, and fixed-size executor across OPA authorisation requests.

Changes

  • Added a shared PooledHttpClient implementation.

  • Configured a fixed-size daemon-thread executor.

  • Added configurable HTTP settings (

    • OPA_HTTP_POOL_SIZE, default 32
    • OPA_HTTP_CONNECT_TIMEOUT_MS, default 2000
    • OPA_HTTP_REQUEST_TIMEOUT_MS, default 5000
  • Preserved the URI reconstruction behavior used by the upstream OPA HTTP client.

  • Added validation for invalid or non-positive configuration values /defensive

  • Removed shared dumpCache state so cache invalidation is scoped to the current OPA response.

  • Improved exception handling and fail-closed behavior.

  • Marked variables and fields as final where reassignment is not expected.

Configuration

Existing deployments do not need to add configuration because defaults are provided

Example:

OPA_HTTP_POOL_SIZE=32
OPA_HTTP_CONNECT_TIMEOUT_MS=2000
OPA_HTTP_REQUEST_TIMEOUT_MS=5000

At startup, the authoriser logs the active HTTP client configuration:

Initialized shared OPA HttpClient (poolSize=32, connectTimeout=PT2S, requestTimeout=PT5S)

@Jimvin Jimvin requested a review from a team June 30, 2026 05:16
@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jun 30, 2026

@NickLarsenNZ NickLarsenNZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this on the grounds that @Jimvin tested it.

He did ask:

What would be nice would be a way to configure the number of threads from the nificluster CR. At the moment the thread pool size is 32 by default, tuning this would be helpful.

@NickLarsenNZ NickLarsenNZ moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jul 2, 2026
@NickLarsenNZ NickLarsenNZ merged commit 2c5b593 into stackabletech:main Jul 2, 2026
1 check passed
@NickLarsenNZ NickLarsenNZ moved this from Development: In Review to Development: Done in Stackable Engineering Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Development: Done

Development

Successfully merging this pull request may close these issues.

3 participants