[#37930][Runners] Reap child processes in ProcessManager to prevent zombie accumulation#37932
Conversation
…vent zombie accumulation ProcessManager.stopProcess() calls destroy()/destroyForcibly() to terminate child processes, but never calls Process.waitFor() to collect the exit status. On POSIX systems, this means the terminated child process entry remains in the kernel process table as a zombie (state Z/defunct) until the parent process itself exits. In long-running environments like Flink TaskManagers using --environment_type=PROCESS, the expansion service processes are repeatedly spawned and stopped but never reaped. Over time this leads to significant zombie accumulation (176+ observed on production hosts). The fix adds process.waitFor() calls after process termination in: - stopProcess(): after the destroy/destroyForcibly sequence - killAllProcesses(): after destroyForcibly in the shutdown hook path Fixes apache#37930
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical issue where child processes were not being properly reaped, leading to an accumulation of zombie processes in long-running Flink TaskManagers. By adding Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
Abacn
left a comment
Abacn
left a comment
There was a problem hiding this comment.
Thanks for reporting this issue and the fix!
|
Different tests failed in two attempts Run 1 (https://github.com/apache/beam/runs/68383107343): testHotKeyLoggingNotEnabled[0: [streamingEngine=false]] (org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest) failed testBasic[0: [streamingEngine=false]] (org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest) failed testHotKeyLogging[0: [streamingEngine=false]] (org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest) failed testBasicHarness[0: [streamingEngine=false]] (org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest) failed All are StreamingDataflowWorkerTest cc: Run 2: testMultipleClientsFailingIsHandledGracefullyByServer (org.apache.beam.runners.fnexecution.logging.GrpcLoggingServiceTest) failed Not related to this change. Merging for now. |
2 participants
Summary
Fixes #37930
ProcessManager.stopProcess() calls destroy()/destroyForcibly() but never Process.waitFor(), so terminated child processes remain as zombies in the kernel process table.
On long-running Flink TaskManagers with --environment_type=PROCESS, expansion service processes accumulate as zombies over time (176+ observed on production hosts). Container-level init systems (dumb-init, tini) cannot help because
the zombies are children of the still-running TaskManager — only the parent can reap them.
Changes
Adds process.waitFor() calls in:
Testing
Existing ProcessManagerTest suite passes. The zombie issue is only observable at the OS level over time and cannot be meaningfully unit-tested — it was confirmed by monitoring production Flink pods.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md