Skip to content

pipeline destroy do not rely on stop propagate signal - #161

Merged
b3by merged 12 commits into
meetecho:mainfrom
GaijinKa:hotfix/pipeline-destroy-do-not-rely-on-stop-propagate-signal
Apr 17, 2026
Merged

pipeline destroy do not rely on stop propagate signal#161
b3by merged 12 commits into
meetecho:mainfrom
GaijinKa:hotfix/pipeline-destroy-do-not-rely-on-stop-propagate-signal

Conversation

@GaijinKa

Copy link
Copy Markdown
Member

Description

This PR refines the lifecycle management of nodes and pipelines within the Juturna core library. The primary focus is on implementing a robust shutdown mechanism that ensures data integrity by "draining" nodes before they fully stop. By introducing a join() method and using threading conditions to track pending updates, the system now waits for active tasks to complete before releasing resources.

These changes address several race conditions and resource management issues during pipeline termination, specifically targeting the improvements discussed in issue #160.

PR type

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test update
  • Build/CI configuration change
  • Other (please describe):

Key modifications and changes

  • Enhanced Thread Safety & Synchronization:
    • Added a _pending_condition (threading.Condition) and a _pending_updates counter to the Node class to track in-flight data processing.
    • Implemented a join() method in Node to safely wait for internal threads (source, worker, update) to terminate, ensuring they are not joined by the current thread.
    • Fixed a bug in the synchroniser setter where it incorrectly assigned to the public property name instead of the private _synchroniser attribute.
  • Improved Shutdown Logic:
    • Introduced a _draining event to signal when a node is stopping, allowing it to discard new incoming messages while finishing current work.
    • Modified Pipeline.stop() to iterate through nodes layer-by-layer (BFS) and explicitly call join() on each node after sending a STOP signal.
    • Updated Buffer.get and internal node loops to use timeouts (0.1s) instead of blocking indefinitely, allowing threads to check for stop events more frequently.
  • Message & Signal Handling:
    • Fix a bug where node communication for SUSPEND and RESUME signals were not wrapped in a Message and ControlPayload objects.
  • State & Resource Management:
    • Added DESTROYED to PipelineStatus and ensured the pipeline status is updated upon calling the destroy() method.
    • Simplified PipelineManager.stop_pipeline by removing redundant manual status updates, delegating state management to the pipeline itself.

Affected components

  • Juturna Core Library:
    • juturna.components (_buffer.py, _node.py, _pipeline.py, _pipeline_manager.py)

Additional context

The transition from blocking get() calls to timeout-based loops significantly reduces the risk of "zombie" threads remaining active after a pipeline stop command. By tracking _pending_updates with a Condition variable, the node now ensures that any call to the update() method is fully completed before the thread is joined and resources are released.

- Implement "draining" state to prevent message loss during shutdown.
- Use threading.Condition to ensure all pending updates complete before joining.
- Replace sentinel None values in queues with timeouts to prevent hung threads.
- Fix potential deadlock by checking current_thread during join.
- Add 'Dumper' test sink and updated pipeline configurations for validation.
@GaijinKa
GaijinKa requested a review from b3by April 15, 2026 14:37
@GaijinKa GaijinKa changed the title Hotfix/pipeline destroy do not rely on stop propagate signal pipeline destroy do not rely on stop propagate signal Apr 15, 2026
- add e2e test to check draining mode
- add e2e test to check immediate stop
- add unit test to check message overflow resilience
- add unit test to check deadlock resilience on real work
@b3by
b3by merged commit d8faffe into meetecho:main Apr 17, 2026
2 checks passed
@GaijinKa
GaijinKa deleted the hotfix/pipeline-destroy-do-not-rely-on-stop-propagate-signal branch April 20, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants