Skip to content

[To dev/1.3] [Pipe] Fix processor worker starvation on pipe stop (#18396) - #18443

Open
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-3c0c7fef-dev-1.3
Open

[To dev/1.3] [Pipe] Fix processor worker starvation on pipe stop (#18396)#18443
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-3c0c7fef-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Cherry-pick 3c0c7fef0068162d7b3fc582df61a82050382ac8 from master to dev/1.3.

Original PR: #18396

  • Add an epoch-based execution guard so a processor invocation that observes STOP yields even if the Pipe is immediately started again.
  • Check the guard around TsFile-close waits, parser admission, tablet boundaries, and OOM backoff while preserving the current event and parsing progress.
  • Make parser admission non-blocking for processor subtasks so one waiting Pipe does not starve another Pipe sharing the same worker.
  • Cancel pending parser requests on STOP and release active parser reservations before yielding.
  • Watch processor workers for continuously running events and emit bounded, rate-limited stack diagnostics.

Backport adaptations

  • Wire the scheduled worker watcher through the dev/1.3 subtask executor API because that scheduler support predates this change on master.
  • Retain the dev/1.3 inline logging style instead of pulling in the newer i18n message layer.
  • Adapt the new tests to the dev/1.3 Pipe pattern, TsFile, and Java 8 APIs.

Verification

  • Spotless: passed for iotdb-core/node-commons and iotdb-core/datanode.
  • Java 8 main/test compilation: passed for both affected modules; Checkstyle reported 0 violations.
  • Focused tests: 23 passed, 0 failures:
    • PipeProcessorSubtaskWorkerTest
    • PipeProcessorSubtaskExecutionGuardTest
    • PipeTsFileInsertionEventAdmissionTest
    • PipeMemoryManagerTest
    • PipeProcessorSubtaskExecutorTest

This PR has:

  • been self-reviewed.
  • added or updated unit tests.

* Fix processor worker starvation on pipe stop

* Add multi-pipe processor worker test

* Log long-running pipe processor event stacks

* refactor(pipe): reuse processor exception root cause
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant