Skip to content

Fix PubSub replay value retention - #6805

Merged
tim-smart merged 3 commits into
mainfrom
agent/codex-engineer/7775a449
Jul 31, 2026
Merged

Fix PubSub replay value retention#6805
tim-smart merged 3 commits into
mainfrom
agent/codex-engineer/7775a449

Conversation

@tim-smart

@tim-smart tim-smart commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • store each subscription's replay values in a bounded snapshot instead of retaining a cursor into the append-only replay list
  • preserve sliding PubSub ordering with a bounded ring of values removed from active subscriptions
  • release replay snapshots when drained or unsubscribed
  • add deterministic object-graph regressions for drained and never-draining replay windows

Root cause

ReplayWindowImpl retained a node from ReplayBuffer's linked list. Future publications extended that same list, so a live subscription could keep every later value reachable even after its replay window was drained. A never-draining subscription retained the same unbounded suffix from its original replay node.

The replay window cannot simply derive every take from the current buffer head: existing replay semantics require a subscription to retain its original replay snapshot, while sliding PubSubs replace only values that are explicitly slid out. This change keeps those semantics while bounding all replay-owned references.

Validation

  • pnpm lint-fix
  • pnpm test --run packages/effect/test/PubSub.test.ts packages/effect/test/SubscriptionRef.test.ts (38 tests)
  • pnpm check

Closes EFF-262
Closes EFF-264
Closes #6804

@github-project-automation github-project-automation Bot moved this to Discussion Ongoing in PR Backlog Jul 31, 2026
@changeset-bot

changeset-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9837b66

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/vitest Patch
@effect/ai-anthropic Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-slopcop effect-slopcop Bot added 4.0 bug Something isn't working labels Jul 31, 2026
Comment thread packages/effect/src/PubSub.ts Outdated
Comment thread packages/effect/test/PubSub.test.ts Outdated

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • ReplayWindowImpl refactored to use bounded arrays — snapshots replay values at construction time and releases references on drain/unsubscribe, breaking the unbounded retention chain.
  • ReplayBuffer.slide() stores slid-out values — a bounded ring buffer feeds replay windows during sliding PubSub operation without holding the growing linked list.
  • PubSub slide() methods pass slid-out values — all four implementations now capture and forward the slid-out value, and slide() is called only when a value actually slides out.
  • Three new deterministic regression tests — object-graph reachability checks for drained and undrained windows, plus replay order preservation across multiple slides.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@tim-smart
tim-smart force-pushed the agent/codex-engineer/7775a449 branch from 6a40234 to 8a9f103 Compare July 31, 2026 03:00

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Index-based replay orderingReplayBuffer.offer() returns a publisher index; each PubSub implementation stores per-slot replay indices; slideValues entries carry {value, index} pairs so ReplayWindowImpl.sync() deduplicates by index rather than rebuilding the window.
  • close() promoted to the ReplayWindow interface — removes the instanceof ReplayWindowImpl check in unsubscribe.
  • Simplified sync() logic — single unified loop with index comparison replaces the two-branch partial-rotation / full-rebuild approach.
  • New sliding order test — verifies monotonic ordering for a lagging subscriber in a scoped sliding pubsub.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
basic.ts 6.63 KB 6.63 KB 0.00 KB (0.00%)
batching.ts 9.42 KB 9.42 KB 0.00 KB (0.00%)
brand.ts 6.31 KB 6.31 KB 0.00 KB (0.00%)
cache.ts 10.12 KB 10.12 KB 0.00 KB (0.00%)
config.ts 20.04 KB 20.04 KB 0.00 KB (0.00%)
differ.ts 19.93 KB 19.93 KB 0.00 KB (0.00%)
http-client.ts 20.97 KB 20.97 KB 0.00 KB (0.00%)
logger.ts 10.28 KB 10.28 KB 0.00 KB (0.00%)
metric.ts 8.55 KB 8.55 KB 0.00 KB (0.00%)
optic.ts 7.33 KB 7.33 KB 0.00 KB (0.00%)
pubsub.ts 14.41 KB 14.26 KB +0.16 KB (+1.09%)
queue.ts 11.09 KB 11.09 KB 0.00 KB (0.00%)
schedule.ts 10.27 KB 10.27 KB 0.00 KB (0.00%)
schema-class.ts 18.86 KB 18.86 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 28.67 KB 28.67 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 24.99 KB 24.99 KB 0.00 KB (0.00%)
schema-string-transformation.ts 12.95 KB 12.95 KB 0.00 KB (0.00%)
schema-string.ts 10.65 KB 10.65 KB 0.00 KB (0.00%)
schema-template-literal.ts 14.85 KB 14.85 KB 0.00 KB (0.00%)
schema-toArbitraryLazy.ts 21.66 KB 21.66 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 24.08 KB 24.08 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 18.91 KB 18.91 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 18.73 KB 18.73 KB 0.00 KB (0.00%)
schema-toFormatter.ts 18.59 KB 18.59 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 22.33 KB 22.33 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 19.25 KB 19.25 KB 0.00 KB (0.00%)
schema.ts 18.12 KB 18.12 KB 0.00 KB (0.00%)
stm.ts 12.05 KB 12.05 KB 0.00 KB (0.00%)
stream.ts 9.37 KB 9.37 KB 0.00 KB (0.00%)

@tim-smart

Copy link
Copy Markdown
Contributor Author

Benchmark rerun for 9837b66b9 on Node v24.18.0.

Harness: makeAtomicUnbounded({ replay: 1 }), one draining subscriber, publish + poll + slide, 5 alternating old/fixed process pairs, 5 timed rounds per process, 5,000,000 operations per round. Per-process minima:

  • 7f0d6ffc1: 18.32, 18.41, 18.71, 18.49, 18.62 ns/op (mean 18.51 ns/op)
  • 9837b66b9: 17.68, 17.51, 17.72, 17.56, 17.84 ns/op (mean 17.66 ns/op)

That is a measured reduction of 0.85 ns/op (4.6%) in this fresh run. The absolute timings are lower than the earlier local experiment, but all paired runs consistently favored the stable node shape.

Validation:

  • pnpm test --run packages/effect/test/PubSub.test.ts packages/effect/test/SubscriptionRef.test.ts — 38 tests passed
  • pnpm check — passed

@tim-smart
tim-smart merged commit 9716990 into main Jul 31, 2026
18 checks passed
@tim-smart
tim-smart deleted the agent/codex-engineer/7775a449 branch July 31, 2026 04:35
@github-project-automation github-project-automation Bot moved this from Discussion Ongoing to Done in PR Backlog Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.0 bug Something isn't working

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

PubSub replay window pins the replay list — SubscriptionRef leaks every published value

1 participant