Skip to content

fix: break circular GObject references that leaked pipelines and sockets#465

Merged
srperens merged 6 commits intomainfrom
fix/pipeline-reference-leak
Mar 30, 2026
Merged

fix: break circular GObject references that leaked pipelines and sockets#465
srperens merged 6 commits intomainfrom
fix/pipeline-reference-leak

Conversation

@srperens
Copy link
Copy Markdown
Collaborator

@srperens srperens commented Mar 30, 2026

Summary

  • Strong references to pipeline and elements captured in connect_pad_added closures in linking.rs created a circular reference chain: pipeline owns elements → elements own signal handler closures → closures own pipeline+elements. This prevented GStreamer from ever finalizing the pipeline, leaking all OS resources (UDP sockets, threads, file descriptors) on every pipeline restart.
  • On production, this accumulated ~5870 leaked UDP sockets after repeated flow restarts, causing kernel socket buffer overflow and AES67 packet loss (21% loss).
  • Replaced strong refs with WeakRef in dynamic pad handler closures, balanced add_signal_watch/remove_signal_watch ref-counting in bus cleanup, and added permanent runtime verification + regression test.

Changes

  • linking.rs — Replace pipeline.clone() and elements.clone() with WeakRef in setup_dynamic_pad_handlers() closures
  • bus.rs — Balance remove_signal_watch calls to match add_signal_watch count
  • mod.rs — Clean up Drop impl ordering (stop QoS task before set_state)
  • state.rs — Add runtime leak detection in stop_flow(): logs ERROR if pipeline/elements survive after PipelineManager drop
  • state.rs — Add flow_name(), element_weak_refs() accessors
  • CLAUDE.md — Add rule: never capture strong GStreamer refs in signal handler closures
  • pipeline_lifecycle_test.rs — Permanent regression test using real PipelineManager

Test plan

  • cargo test --test pipeline_lifecycle_test passes (weak refs verify full finalization)
  • Tested locally: 58 flows started+stopped, zero leaked elements in log
  • Deployed to production: flow restart, socket count stable (no growth), zero ERROR in logs
  • All block-level closures audited — all use WeakRef or safe types (no strong GStreamer captures)

🤖 Generated with Claude Code

Per Enstedt and others added 6 commits March 30, 2026 10:41
Strong references to pipeline and elements captured in connect_pad_added
closures in linking.rs created a circular reference: pipeline owns elements,
elements own signal handler closures, closures own pipeline+elements.
This prevented GStreamer from ever finalizing the pipeline, leaking all
OS resources (UDP sockets, threads) on every pipeline restart.

On production (br), this accumulated ~5870 leaked UDP sockets after
repeated flow restarts, causing kernel socket buffer overflow and
AES67 packet loss.

Changes:
- Replace strong refs with WeakRef in dynamic pad handler closures
- Balance add_signal_watch/remove_signal_watch calls in bus watch cleanup
- Add permanent runtime verification in stop_flow() that logs ERROR if
  pipeline or elements survive after PipelineManager drop
- Add regression test that creates a real pipeline via PipelineManager
  and asserts full GObject finalization after stop+drop

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Never capture strong references to gst::Pipeline, gst::Element, or
gst::Bin inside signal handler closures — this creates circular
references that prevent pipeline finalization and leak OS resources.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ences

Intentionally creates a strong pipeline ref inside a signal handler
closure to verify that the weak ref detection mechanism works. If this
test ever fails, it means real leaks would go undetected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip info.version before comparing so patch-level version bumps
without API changes don't fail the snapshot test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI runner lacks gstreamer1.0-plugins-good (no 'level' element).
Simplify test flow to audiotestsrc → fakesink which only needs
base plugins.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@srperens srperens merged commit 1eec951 into main Mar 30, 2026
7 checks passed
@srperens srperens deleted the fix/pipeline-reference-leak branch March 30, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant