Fixes #30491: Bound async DB fan-out and prepare JDK 25 runtime#30492
Fixes #30491: Bound async DB fan-out and prepare JDK 25 runtime#30492harshach wants to merge 16 commits into
Conversation
github-actions
Bot
added
backend
safe to test
There was a problem hiding this comment.
Pull request overview
This PR introduces a bounded “DB-heavy” async execution lane to prevent unbounded fan-out from exhausting the shared HikariCP pool, while also exposing configurable background executor limits and preparing Alpine-based server images for a JDK 25 runtime (keeping Java 21 bytecode and virtual threads disabled by default).
Changes:
- Add a bounded async executor view for top-level DB-heavy operations (with Micrometer gauges for active/queued/limit) and switch key async entry points to use it.
- Add explicit configuration blocks for async DB limits and background executor parallelism/lanes/permits/workers; initialize these budgets early during app startup.
- Upgrade HikariCP to 7.1.0 and update Alpine runtime images to install
openjdk25-jre.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Bumps HikariCP version to 7.1.0. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java | Adds a semaphore-gated ExecutorService view for bounded concurrency. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/AsyncService.java | Adds bounded executor lane, config-driven initialization, and Micrometer gauges. |
| openmetadata-service/src/main/java/org/openmetadata/service/security/auth/UserActivityTracker.java | Makes DB permit semaphore configurable and adds explicit initialization. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java | Routes reindex worker to bounded lane; keeps watchdog on raw lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java | Routes lineage export async work to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/glossary/GlossaryTermResource.java | Routes glossary move async work to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/EntityResource.java | Routes async delete/restore and bulk async ops to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/columns/ColumnResource.java | Routes async column import/update to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/apps/AppResource.java | Routes async app delete to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/ai/AuditPackGenerator.java | Routes audit pack generation to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplicationConfig.java | Adds new config sections: asyncOperations and backgroundExecutors. |
| openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java | Initializes async/background budgets early and wires configurable background job workers. |
| openmetadata-service/src/main/java/org/openmetadata/service/jobs/GenericBackgroundWorker.java | Makes background job worker pool size configurable. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestSuiteRepository.java | Routes async test suite deletion to bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/OrderedLaneExecutor.java | Adds configurable lane count support (0 = auto). |
| openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java | Initializes ordered lanes with configurable lane count. |
| openmetadata-service/src/main/java/org/openmetadata/service/events/EventFilter.java | Makes change-event ForkJoinPool parallelism configurable. |
| openmetadata-service/src/main/java/org/openmetadata/service/config/BackgroundExecutorsConfiguration.java | New config class for background executor limits. |
| openmetadata-service/src/main/java/org/openmetadata/service/config/AsyncOperationsConfiguration.java | New config class for bounded async DB-heavy operations. |
| openmetadata-service/src/test/java/org/openmetadata/service/util/BoundedAsyncExecutorTest.java | Adds unit tests for bounded executor semantics. |
| openmetadata-service/src/test/java/org/openmetadata/service/util/AsyncServiceTest.java | Adds test coverage for bounded-vs-raw executor behavior. |
| openmetadata-service/src/test/java/org/openmetadata/service/jobs/GenericBackgroundWorkerTest.java | Adds test for configurable worker pool sizing. |
| openmetadata-service/src/test/java/org/openmetadata/service/events/lifecycle/OrderedLaneExecutorTest.java | Adds test for configured lane-count override behavior. |
| openmetadata-service/src/test/java/org/openmetadata/service/config/LoggingConfigurationYamlTest.java | Adds openmetadata-h2-test.yaml to logging YAML validation list. |
| openmetadata-service/src/test/java/org/openmetadata/service/config/BackgroundExecutorsConfigurationTest.java | Adds tests for new background/async config defaults. |
| docker/docker-compose-quickstart/Dockerfile | Switches Alpine runtime to install openjdk25-jre. |
| docker/development/Dockerfile | Switches Alpine runtime to install openjdk25-jre. |
| DEVELOPER.md | Documents per-pod database connection budgeting and relevant config keys. |
| conf/openmetadata.yaml | Adds asyncOperations/backgroundExecutors config blocks; cleans up DB pool comments/keys. |
| conf/openmetadata-h2-test.yaml | Mirrors new config blocks and DB pool comment cleanup for test config. |
| public static synchronized AsyncService getInstance() { | ||
| if (instance == null) { | ||
| instance = new AsyncService(); | ||
| LOG.warn("AsyncService not initialized, using defaults"); | ||
| initialize(new AsyncOperationsConfiguration()); | ||
| } |
| public static synchronized void initialize(AsyncOperationsConfiguration config) { | ||
| if (instance == null) { | ||
| instance = new AsyncService(config); | ||
| instance.registerMetrics(); | ||
| } | ||
| } |
| private static int resolveLaneCount(int configuredLaneCount) { | ||
| int resolvedLaneCount = configuredLaneCount; | ||
| if (resolvedLaneCount == 0) { | ||
| resolvedLaneCount = computeAutomaticLaneCount(); | ||
| } | ||
| return resolvedLaneCount; | ||
| } |
🔴 Playwright Results — workflow failedValidated commit ✅ 1187 passed · ❌ 1 failed · 🟡 1 flaky · ⏭️ 15 skipped · 🧰 0 lifecycle flaky Pipeline and setup failures (3)
PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 56m 49s ⏱️ Max setup 3m 33s · max shard execution 21m 0s · max shard-job elapsed before upload 24m 48s · reporting 8s 🌐 233.22 requests/attempt · 2.31 app boots/UI scenario · 23.41% common-shard skew Optimization targets still in progress:
Genuine Failures (failed on all attempts)❌
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java:1103
- The watchdog timeout starts counting immediately after submission, but the bounded executor may spend significant time queued waiting for a permit. That means a reindex job can be cancelled before it ever starts running, undermining the new “queues without rejection” behavior. Consider measuring the timeout from when the worker actually begins (e.g., CountDownLatch/CompletableFuture signaled at the top of the worker) or using a two-phase timeout: wait-for-start + run-timeout.
// The watchdog waits on the worker future and must not consume a DB-task permit.
AsyncService.getInstance()
.getExecutorService()
.submit(
() -> {
try {
future.get(timeoutMinutes, TimeUnit.MINUTES);
} catch (TimeoutException e) {
future.cancel(true);
LOG.error(
"Reindex job timed out after {} minutes and was cancelled", timeoutMinutes);
} catch (Exception e) {
| private void runWithPermit(Runnable command) { | ||
| try { | ||
| semaphore.acquire(); | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| return; | ||
| } | ||
| try { | ||
| command.run(); | ||
| } finally { | ||
| semaphore.release(); | ||
| } | ||
| } |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java:52
runWithPermitdrops non-FutureRunnabletasks if the permit wait is interrupted (it just returns). That can silently lose work in cases likeshutdownNow()interrupting a virtual thread blocked onSemaphore.acquire(), and more generally violates the expectation thatexecute()tasks either run or are explicitly rejected/cancelled.
Consider looping on InterruptedException and only exiting early when the delegate is shutting down or when the wrapped command is a cancelled Future. Preserve the interrupted flag before returning/running so callers can observe it if needed.
private void runWithPermit(Runnable command) {
try {
semaphore.acquire();
} catch (InterruptedException e) {
cancelIfFuture(command);
Thread.currentThread().interrupt();
return;
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 42 out of 42 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java:51
- If the delegate interrupts a task while it's blocked on permit acquisition,
runWithPermitreturns and the task never runs. Forsubmit(...)this is surfaced as a cancelled Future, but for plainexecute(Runnable)it's currently silent; adding an explicit note here will help avoid accidental misuse ofexecute(...)for important work.
| asyncOperations: | ||
| # Max concurrently RUNNING DB-heavy async tasks (async delete/restore, bulk tag/asset | ||
| # ops, glossary move, column/lineage CSV, app delete, audit packs). Excess tasks queue | ||
| # (parked virtual threads, FIFO) — nothing is rejected. 0 = unbounded. | ||
| maxConcurrentDbTasks: ${ASYNC_OPERATIONS_MAX_CONCURRENT_DB_TASKS:-25} |
| page.getByTestId('searchBox').press('Enter'), | ||
| ]); | ||
| hasSubmittedSearch = true; | ||
| await submitSearch(crypto.randomUUID().replaceAll('-', '')); |
| static void submit(UUID reportId) { | ||
| AsyncService.getInstance().execute(() -> run(reportId)); | ||
| AsyncService.getInstance().getBoundedExecutorService().submit(() -> run(reportId)); | ||
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 42 out of 42 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
openmetadata-service/src/main/java/org/openmetadata/service/resources/ai/AuditPackGenerator.java:86
submit(...)returns aFuturethat is ignored here, which adds overhead (creates aFutureTask) and introduces cancellation semantics that nobody observes. Since this is fire-and-forget dispatch, preferexecute(...)(or store/handle theFutureif cancellation/report-state handling is intended).
static void submit(UUID reportId) {
AsyncService.getInstance().getBoundedExecutorService().submit(() -> run(reportId));
}
openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java:28
getQueuedCount()currently relies onSemaphore#getQueueLength(), which is an estimate. Add an explicit queued-task counter so the queue gauge reflects the actual number of tasks waiting for a permit.
openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java:49- Permit-wait interruption should decrement the queued counter to avoid permanently inflating the exported queue metric, and successful acquisition should decrement before running the command.
openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java:72 - Queue size should be derived from the explicit queued-task counter rather than
Semaphore#getQueueLength()to keep the metric deterministic.
| import java.util.concurrent.AbstractExecutorService; | ||
| import java.util.concurrent.ExecutorService; | ||
| import java.util.concurrent.Future; | ||
| import java.util.concurrent.Semaphore; | ||
| import java.util.concurrent.TimeUnit; |
| Entity.getCollectionDAO() | ||
| .relationshipDAO() | ||
| .insert( | ||
| broken.getTestDefinition().getId(), | ||
| broken.getId(), | ||
| Entity.TEST_DEFINITION, | ||
| Entity.TEST_CASE, | ||
| Relationship.CONTAINS.ordinal()); |
| JsonNode node = findNode(fetchGlossaryGraph(glossaryId), termId); | ||
| assertNotNull(node.get("id"), () -> "Term " + termId + " should be projected to RDF"); | ||
| assertEquals( | ||
| glossaryId.toString(), | ||
| node.path("glossaryId").asText(null), | ||
| () -> "Term " + termId + " should carry its RDF glossary projection"); |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/search/TestCaseStaleDefinitionReindexIT.java:121
- cleanupBrokenTestCase() re-inserts the CONTAINS relationship in the opposite direction (TEST_DEFINITION -> TEST_CASE) from what was deleted (TEST_CASE -> TEST_DEFINITION). EntityRelationshipDAO.insert(fromId, toId, fromEntity, toEntity, ...) is directional, so this does not restore the broken relationship and can leave cleanup incomplete/flaky.
Entity.getCollectionDAO()
.relationshipDAO()
.insert(
broken.getTestDefinition().getId(),
broken.getId(),
Entity.TEST_DEFINITION,
Entity.TEST_CASE,
Relationship.CONTAINS.ordinal());
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 45 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/RdfGlossaryGraphIT.java:654
awaitTermInGraphcan throw a NullPointerException whenfindNode(...)returns null:node.get("id")is dereferenced before asserting the node exists. This makes the Awaitility poll fail with an exception instead of retrying until the term appears in the RDF graph.
JsonNode node = findNode(fetchGlossaryGraph(glossaryId), termId);
assertNotNull(node.get("id"), () -> "Term " + termId + " should be projected to RDF");
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 46 out of 46 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java:1133
awaitExecutionStartOrCompletion(...)polls indefinitely with no upper bound. Once the reindex worker is submitted to the bounded executor, admission can be delayed arbitrarily under sustained saturation; this helper should enforce a timeout (and propagateTimeoutException) so the watchdog can cancel queued work.
static void awaitExecutionStartOrCompletion(CountDownLatch executionStarted, Future<?> future)
throws InterruptedException {
while (executionStarted.getCount() > 0 && !future.isDone()) {
executionStarted.await(EXECUTION_START_POLL_MILLIS, TimeUnit.MILLISECONDS);
}
}
openmetadata-service/src/test/java/org/openmetadata/service/resources/search/SearchResourceAsyncTest.java:92
- This test calls
SearchResource.awaitExecutionStartOrCompletion(...); with the production fix adding a timeout parameter, update the test invocation to pass a deadline and keep the intent (exit promptly once the future is cancelled).
| () -> { | ||
| try { | ||
| awaitExecutionStartOrCompletion(executionStarted, future); | ||
| future.get(timeoutMinutes, TimeUnit.MINUTES); | ||
| } catch (TimeoutException e) { |
| Future<?> watchdog = | ||
| watchdogExecutor.submit( | ||
| () -> { | ||
| watchdogStarted.countDown(); | ||
| SearchResource.awaitExecutionStartOrCompletion(executionStarted, reindexTask); | ||
| reindexTask.get(100, TimeUnit.MILLISECONDS); | ||
| return null; | ||
| }); |
| private void registerMetrics() { | ||
| Gauge.builder(ACTIVE_TASKS_METRIC, this, AsyncService::getActiveDbTaskCount) | ||
| .description("Running DB-heavy asynchronous operations") | ||
| .register(Metrics.globalRegistry); | ||
| Gauge.builder(QUEUED_TASKS_METRIC, this, AsyncService::getQueuedDbTaskCount) | ||
| .description("Queued DB-heavy asynchronous operations") | ||
| .register(Metrics.globalRegistry); | ||
| Gauge.builder(TASK_LIMIT_METRIC, this, service -> service.maxConcurrentDbTasks) | ||
| .description("Configured DB-heavy asynchronous operation limit") | ||
| .register(Metrics.globalRegistry); | ||
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 50 out of 50 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java:1105
- The watchdog waits indefinitely in
awaitExecutionStartOrCompletion(...)before starting thefuture.get(timeoutMinutes, ...)timer. Under saturation of the bounded DB lane, this allows reindex requests to exceed the configured timeout by an arbitrary admission delay (and keeps an extra watchdog task alive the whole time). Consider enforcing a single end-to-end deadline that includes permit/admission wait, cancelling the job if it cannot start within the configured timeout.
try {
awaitExecutionStartOrCompletion(executionStarted, future);
future.get(timeoutMinutes, TimeUnit.MINUTES);
openmetadata-ui/src/main/resources/ui/playwright/utils/domain.ts:300
- Same issue as
assignDomainWidget: relying onclickAvailableWidgetAction(addBtn, editBtn)can wait on a hidden DOM-first button and time out even though the other action is visible. Prefer clicking the first visible button.
| const editBtn = page.getByTestId('edit-domain'); | ||
| const isAdd = await addBtn.isVisible(); | ||
| await (isAdd ? addBtn : editBtn).click(); | ||
| await clickAvailableWidgetAction(addBtn, editBtn); |
|
Code Review 👍 Approved with suggestions 2 resolved / 3 findingsBounds top-level asynchronous database operations and prepares the Alpine runtime for JDK 25. Consider correcting the jackson-bom dependency comment referencing a future CVE identifier. 💡 Quality: Misleading jackson comment: '3.1.5 adds CVE-2026-59889'The comment on the jackson-bom bump reads "Force patched tools.jackson 3.x ... 3.1.5 adds CVE-2026-59889", which literally states the upgrade introduces a vulnerability — contradicting the stated intent of forcing a patched version. This will mislead future maintainers about why the version is pinned. Reword to say 3.1.5 fixes/addresses CVE-2026-59889. Clarify that 3.1.5 remediates the CVE rather than introducing it.✅ 2 resolved✅ Edge Case: Reindex timeout now counts bounded-lane queue wait time
✅ Quality: Cleanup in finally can throw and mask test failure
🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| } catch (TimeoutException e) { | ||
| future.cancel(true); | ||
| LOG.error( | ||
| "Reindex job timed out after {} minutes and was cancelled", timeoutMinutes); |
There was a problem hiding this comment.
When a reindex operation exceeds its execution timeout during a blocking database or search operation, future.cancel(true) interrupts the worker, but the per-entity catch (Exception) handles the interruption-related failure as an ordinary entity failure and continues the loop. Database and search writes can therefore continue after the watchdog reports that the job was cancelled.
Knowledge Base Used: Search Indexing and Event/Notification Sync
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 48 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java:1106
- The watchdog can block indefinitely in awaitExecutionStartOrCompletion() if the reindex worker never gets admitted to the bounded executor (e.g., persistent backlog). This makes the effective timeout unbounded under load. Consider bounding the admission wait (e.g., by timeoutMinutes) so the watchdog eventually cancels the job rather than waiting forever.
() -> {
try {
awaitExecutionStartOrCompletion(executionStarted, future);
future.get(timeoutMinutes, TimeUnit.MINUTES);
} catch (TimeoutException e) {
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java:1133
- awaitExecutionStartOrCompletion() currently waits without any upper bound, so a queued reindex job can leave the watchdog stuck forever (and leak a virtual thread) if it never starts. Add a timed overload and use it from the watchdog to cap admission wait and surface a TimeoutException.
static void awaitExecutionStartOrCompletion(CountDownLatch executionStarted, Future<?> future)
throws InterruptedException {
while (executionStarted.getCount() > 0 && !future.isDone()) {
executionStarted.await(EXECUTION_START_POLL_MILLIS, TimeUnit.MILLISECONDS);
}
}
Describe your changes:
Fixes #30491
I bounded top-level DB-heavy
AsyncServicework because unbounded request fan-out could exhaust the shared HikariCP pool; the change also exposes background executor limits, adds queue metrics and a connection-budget ledger, upgrades HikariCP to 7.1.0, removes unused pool keys, and moves Alpine server runtimes to JDK 25 while keeping Java 21 bytecode andenableVirtualThreadsdisabled by default.Type of change:
High-level design:
The new fair-semaphore executor view acquires permits inside virtual threads, queues excess operations without rejection, and offers
0as an unbounded rollback setting.Internal fire-and-forget work and nested continuations stay on the raw executor to prevent permit deadlocks, while startup injects explicit defaults into the remaining DB-using background executors.
The legacy
minimal-ubuntuimage remains on JDK 21 because its Ubuntu Xenial base cannot provide JDK 25.Tests:
Use cases covered
0restores unbounded behaviorUnit tests
BoundedAsyncExecutorTest,AsyncServiceTest,BackgroundExecutorsConfigurationTest,GenericBackgroundWorkerTest,OrderedLaneExecutorTest, andLoggingConfigurationYamlTestBackend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
mvn spotless:applymvn -q dependency:get -Dartifact=com.zaxxer:HikariCP:7.1.0openjdk25-jrefromalpine:3and verified OpenJDK25.0.3UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.Improvement
Greptile Summary
This PR bounds DB-heavy asynchronous work and updates related runtime configuration.
Confidence Score: 3/5
The PR is not yet safe to merge because a timed-out reindex can continue performing database and search operations after cancellation is reported.
The admission wait is now separated from execution time, but future cancellation only interrupts the worker; its broad per-entity exception handler consumes interruption-related failures and continues processing later entities.
Files Needing Attention: openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java
Important Files Changed
Sequence Diagram
sequenceDiagram participant API as SearchResource participant Bound as Bounded executor participant Worker as Reindex worker participant Watchdog as Timeout watchdog participant DB as Database participant Search as Search backend API->>Bound: submit reindex API->>Watchdog: submit timeout monitor Bound->>Worker: admit after permit Worker->>Watchdog: signal executionStarted loop Each entity Worker->>DB: load entity Worker->>Search: delete and recreate document end Watchdog->>Worker: cancel(true) after timeout Worker-->>Worker: catch exception and continue loopReviews (16): Last reviewed commit: "Merge origin/main into harshach/jetty-vi..." | Re-trigger Greptile
Context used (3)