Prevent integration event and history sync from overwhelming local machines

[kjgbot]

Problem

When a workspace has large integrations or high-volume webhook traffic, Pear can slow down the user's machine. Recent debugging showed several contributing patterns:

• Local historical Relayfile mounts can contain a lot of files and data.
• Polling or recursively watching mounted integration trees can produce a large amount of filesystem work.
• Incoming webhook events can arrive in bursts and currently risk turning into unbounded broker injections.
• Historical data, incoming webhook events, local mounts, and writeback are still too coupled in the runtime model.
• Logging every integration event during a burst can add noticeable overhead.

The desired product behavior is:

• The source/remote Relayfile instance may continue syncing and retaining historical integration data.
• Pear, as a consumer, should not download or mount historical data unless the user explicitly enables historical download.
• Incoming webhook data should still sync remotely, trigger Relayfile events, and notify selected agents without requiring historical local mounts.
• Writeback should remain available without requiring a full local mirror.

Current Direction

Recent PR work made historical local download/mount opt-in in Pear, but there are more architectural hardening tasks before this is safe for high-volume users.

The main fix is not a language rewrite. Rust or Swift may help later for a constrained sidecar, but the first-order issue is unbounded dataflow: polling, recursive watching, broad subscriptions, and event fanout.

Proposed Architecture

1. Event-only default path

Default integration flow should be:

1. Provider webhook arrives in cloud.
2. Cloud/source Relayfile writes the incoming webhook data and emits a scoped event.
3. Pear subscribes to selected Relayfile event paths via the SDK.
4. Pear injects a compact <integration-event> into selected agents.
5. Agents read additional context on demand via mounted history only if historical download is enabled, or via future targeted SDK/API reads.

Pear should not need to mount .integrations/ for event delivery.

2. Historical download is a consumer cache

Historical download/mount should be treated as an optional local cache:

• Default off.
• Explicit user toggle per project/integration/resource.
• Bound by max files, max bytes, max depth, or selected resources where possible.
• Never required for incoming webhook notification.
• Never automatically enabled just because an integration is connected.

3. Writeback should use a small command surface

Writeback should not require recursively watching a full historical tree. Prefer one of:

• SDK writeback API from Pear/agents.
• A tiny .integrations/.outbox or provider-specific command area.
• Targeted file paths only for known writeback-enabled command files.

Avoid recursive watchers on provider roots or history mirrors.

Implementation Tracks

Track A: Remove broad local recursive watchers

• Audit src/main/integration-event-bridge.ts local mount watching.
• Do not call watch(localRoot, { recursive: true }) for full integration history trees.
• Only enable local watching when historical download is on and only for a bounded writeback/outbox command surface.
• Ensure webhook-only mode uses remote Relayfile event stream only.

Acceptance criteria:

• Starting Pear with connected integrations and historical download off creates no recursive watchers for integration data.
• Incoming Slack/GitHub/Linear/Notion webhook events still notify subscribed agents.
• Writeback still works through the intended command/API path.

Track B: Add bounded event queue and backpressure

Add a per-project integration event dispatcher before broker injection:

• Max queued events per project.
• Max injected events per second per project.
• Coalesce by provider + resource path, and for Slack by channel/thread/message path.
• Drop or compact duplicate/low-value events during bursts.
• Emit a summary event when events are compacted, e.g. 12 Slack messages changed in #proj-cloud.
• Avoid Promise.all fanout over large recipient lists without limits.

Acceptance criteria:

• A burst of 1,000 file events does not create 1,000 immediate broker messages.
• CPU remains bounded during bursts.
• Agents receive useful summaries rather than a flood.

Track C: Server-side scoped subscriptions

Make sure Relayfile/cloud only sends relevant changes to Pear:

• Subscribe to exact selected resources, not provider roots.
• Avoid replaying historical changes as new events when Pear starts, unless explicitly requested.
• Support from=now or equivalent resume semantics for default subscriptions.
• Ensure event stream reconnect does not replay large historical backlogs by default.

Acceptance criteria:

• Restarting Pear after a large integration sync does not inject old historical records as fresh events.
• Event subscriptions only receive paths configured in the integration listener UI.

Track D: Lazy context reads

Do not require a local mirror for agents to fetch context:

• Provide a targeted read path for event context, either via Relayfile SDK/API or a small Pear IPC bridge.
• Event payload should include provider, resource path, resource id, title/status/actor when available.
• Agents can request a specific file/resource by path without downloading the whole integration tree.

Acceptance criteria:

• Agent can handle a Slack message event and fetch the specific message/thread context without historical download enabled.
• Fetching context does not mount the provider root locally.

Track E: Recipient and broker fanout optimization

• Cache project agent recipient lists briefly and invalidate when agents change.
• Cache explicit integration notification targets.
• Avoid listing agents for every event during bursts.
• Rate-limit broker sendMessage calls per project.

Acceptance criteria:

• Event bursts do not repeatedly call listAgents for each event.
• Broker message sends are bounded and observable.

Track F: Logging and telemetry budgets

• Gate verbose integration event logs behind a debug flag.
• Aggregate repetitive event stream errors and polling fallback logs.
• Add basic counters for events received, events injected, events coalesced, events dropped, queue depth, and mount count.

Acceptance criteria:

• A high-volume event stream does not spam the main-process console.
• We can diagnose bottlenecks from counters without enabling verbose logs.

Track G: Mount resource budgets

For historical download/mount when explicitly enabled:

• Add warnings before mounting large integrations.
• Enforce or expose limits: max files, max bytes, selected resources only.
• Prefer narrow resource mounts over provider roots.
• Avoid mode: poll for high-volume paths if the SDK can support event-driven or demand-loaded reads.

Acceptance criteria:

• Enabling historical download for a large integration does not silently start an unbounded local sync.
• UI communicates that historical download may be expensive.

Open Questions

• Does @relayfile/sdk support event subscriptions with from=now or a no-backfill cursor? If not, add/track this in Relayfile.
• Can writeback be exposed directly through SDK/API so Pear does not need local filesystem watching for command files?
• Can Relayfile expose targeted resource reads suitable for agents without mounting local history?
• What event compaction semantics should Slack use: per channel, per thread, per message, or time window?

Non-goals

• Rewriting the whole Electron app in Rust or Swift.
• Mounting every connected integration by default.
• Treating historical source sync as disabled. The source remote instance can still retain history; Pear should just avoid downloading it unless requested.

Files To Inspect First

• src/main/integration-event-bridge.ts
• src/main/integration-mounts.ts
• src/main/integrations.ts
• src/main/relayfile-mount-launcher.ts
• Relayfile SDK event subscription and mount APIs

[kjgbot]

Live repro: historical replay flood observed 2026-06-05 ~14:09–14:17 UTC

A Claude Code agent session (claude-1, cloud repo, workspace agentrelayworkspace, Slack channel #proj-cloud / C0AD7UU0J1G) was flooded with relayfile.changed injections — one broker injection per file event, arriving every ~2–15 seconds for several minutes, repeatedly interrupting in-progress turns.

Observed characteristics, mapped to tracks:

Track C (historical replay as fresh events): The majority of injected events were resyncs of messages 11–12 days old (ts 1779625906 … 1780020181) delivered as if they were new activity. Nothing about the payload distinguishes a backfill/reconcile from a live webhook, so the agent has to fetch and inspect each record to discover it's old news.

Track B (no coalescing/backpressure): Dozens of events for the same channel arrived as individual injections. These should have been coalesced into a single summary (e.g. "N Slack messages resynced in #proj-cloud"). The same record was also re-delivered multiple times — one message's meta.json was injected at least 3 times within minutes — and the same logical record was delivered under both the raw channel id path (C0AD7UU0J1G) and the aliased path (C0AD7UU0J1G__proj-cloud), doubling the volume.

Sync-internal artifacts leaked as events: a temp file was injected as a user-relevant event:

/slack/channels/C0AD7UU0J1G/messages/1780666494_028989/.meta.json.tmp-3507823867

Dotfile/*.tmp-* paths should be filtered before injection.

Self-echo loop risk: the agent's own writeback files (e.g. claude-1-codex-spawned.json, claude-1-issue82-ack.json) were re-delivered back to the same agent as incoming integration events. Without filtering, an agent that responds to events by writing writeback files generates new events addressed to itself.

Also observed malformed doubled event paths during the flood — filed separately since it looks like a path-construction defect rather than architecture: see the linked issue.

[kjgbot]

Track A readiness update for #82: PR #98 is ready for review.

PR: #98

Fixed in Track A:

• Disabled Pear local fallback filesystem watching when historical download is off, so connected integrations no longer create recursive local watchers for provider history trees in the default mode.
• Kept webhook/event delivery on the remote Relayfile subscription path; subscribed Slack/GitHub/Linear/Notion-style Relayfile change events still inject <integration-event> notifications to selected agents/channels.
• Restricted the remaining recursive local fallback watcher path to bounded command roots such as .outbox/outbox when historical download is explicitly enabled, preventing broad provider history roots like /slack/channels/... from being watched locally.
• Preserved writeback support through the existing mounted provider/discovery command surface with read/write mount scopes; local writeback draft/command files are filtered from incoming notification injection to avoid self-echo loops.

Acceptance criteria covered:

• Historical download OFF -> zero local recursive watcher roots for integration data trees: covered by localWatchRootsFor(...) regression tests.
• Incoming webhook/Relayfile events still notify subscribed agents/channels: covered by bridge subscription/injection tests.
• Writeback still works through the intended command/API path: covered by integration mount tests preserving provider/discovery write scopes and by filtering local writeback command files from notification injection.

Verification:

• node --experimental-strip-types --no-warnings --test src/main/__tests__/integration-event-bridge.test.ts
• npm test
• npx vitest run src/main/integration-mounts.test.ts
• GitHub Actions on PR Prevent broad integration local event watchers #98 head 3fafd98487bda8a5c8feb5ccadfaa9f4cbe613b9: checks passed, packaged-mcp-smoke passed, CodeRabbit passed.

[kjgbot]

Track C SDK/server contract gap found while implementing server-side scoped subscriptions.

Pear cannot correctly implement the requested default subscription semantics against the current Relayfile SDK/server contract without a client-side workaround.

What I verified:

• Pear pins @relayfile/sdk 0.7.23 (package.json / package-lock.json). I inspected the published @relayfile/sdk@0.7.23 tarball, not just a local Relayfile checkout.
• RelayFileSyncOptions.cursor exists, but in dist/sync.js it is consumed by the HTTP polling loop only. The WebSocket opener constructs /v1/workspaces/:workspaceId/fs/ws?token=... and does not include cursor, from, replayOnStart, pathScope, or any glob/path filter.
• The Relayfile server WebSocket handler subscribes to the workspace and then unconditionally sends GetRecentEvents(workspaceID, 100) as catch-up. There is no query or protocol field for from=now, cursor=<eventId>, replayOnStart=none, or exact path filters.
• RelayFileClient.subscribe(globs, ..., { pathScope }) filters paths client-side after the process has already received workspace-level WebSocket events; it does not create a server-side scoped subscription.

Missing API needed for Track C:

1. A no-backfill WebSocket resume contract, e.g. one of:
   • RelayFileSync({ start: "now" | "no-backfill" }) that maps to /fs/ws?from=now, or
   • RelayFileSync({ cursor }) honored by the WebSocket server as an exclusive resume cursor, including reconnects, or
   • RelayFileClient.subscribe(..., { replayOnStart: "none" }) wired through to the underlying WebSocket connection rather than only the change-log replay cache.
2. Server-side path scoping for subscriptions, e.g. /fs/ws?path=/slack/channels/C.../**&path=/github/repos/.../** or an initial subscribe frame carrying exact globs/path scopes, so Pear does not receive provider-root/workspace-root events and filter them locally.
3. Reconnect semantics must preserve the same no-backfill/path-scope behavior. A reconnect must not trigger the server's current unconditional recent-events catch-up as fresh relayfile.changed events.

I did not fork or patch @relayfile/sdk from Pear. A Pear-only workaround such as forcing polling and locally seeding a cursor could reduce injections, but it would not satisfy the requested server-side scoped/no-backfill subscription contract and would still be the wrong ownership boundary for this issue.

[kjgbot]

Track A merge update for #82: PR #98 has been merged.

PR: #98
Merge commit: 7678929
Merged PR head: 3fafd98
Reviewer verdict: APPROVE from claude-reviewer-1; formal GitHub approve was blocked because the reviewer gh session was authenticated as the PR author account, so the approval was posted as a COMMENTED review and accepted by the coordinator as the merge-gate approval.

Track A fixed:

• Disabled broad local recursive fallback watchers when historical download is off.
• Kept incoming webhook/Relayfile event notification on the remote subscription path, so subscribed integration events still notify agents/channels.
• Restricted local fallback watching to bounded command roots when historical download is explicitly on, with legacy /integrations/<provider>/.outbox command mounts canonicalized to the provider root layout.
• Preserved writeback through the existing provider/discovery mount command surface; writeback draft/command files are filtered from incoming notification injection to avoid self-echo loops.

Acceptance criteria covered:

• Historical download OFF -> no recursive local watcher roots for integration data trees.
• Incoming Slack/GitHub/Linear/Notion-style Relayfile events still notify subscribed agents/channels.
• Writeback remains available through the intended mounted command/schema path; the bridge watcher change does not own writeback upload.

Verification before merge:

• node --experimental-strip-types --no-warnings --test src/main/__tests__/integration-event-bridge.test.ts
• npm test
• npx vitest run src/main/integration-mounts.test.ts
• GitHub Actions: checks passed, packaged-mcp-smoke passed, CodeRabbit passed.

Follow-up note: the reviewer also requested tightening the command-root matcher from .outbox/outbox to dotted .outbox only. I implemented and locally verified that hardening after the merge window; it is staged on branch fix/issue-82-track-a at e6dee90 for either a tiny follow-up PR or inclusion in Track B, pending coordinator direction.

[kjgbot]

Track F PR opened: #100

Covered acceptance:

• Verbose integration event receive/inject/skip logs are quiet by default and gated behind PEAR_INTEGRATION_EVENTS_DEBUG=1.
• Repeated remote stream errors, polling fallback notices, and local watcher failures are aggregated with occurrence/suppression counts.
• Counters for events received/injected/coalesced/dropped, queue depth, and mount count are available via pear.integrations.telemetry().

Validation:

• node --experimental-strip-types --no-warnings --test src/main/tests/integration-event-bridge.test.ts
• npm test
• npm run build

[kjgbot]

Track C Pear-side mitigation PR opened: #103

This is a clearly marked temporary-pending-SDK-contract stopgap while the real Relayfile server/SDK subscription contract lands upstream in AgentWorkforce/relayfile#238.

Covered in PR #103:

• Subscribes only to exact configured resource paths; removed the Slack raw-channel fallback that duplicated selected alias paths.
• Passes current SDK pathScope equal to the exact watch globs so the future server-side path-filter swap is localized.
• Drops remote events whose occurredAt predates the subscription session start when historical download is off.
• Keeps historical replay allowed when downloadHistoricalData is true.
• Leaves local mount events outside the remote session-start gate because local/historical cache behavior is explicit.

Validation:

• node --experimental-strip-types --no-warnings --test src/main/__tests__/integration-event-bridge.test.ts
• npm test (41/41 passing after rebasing onto origin/main at Track A merge 7678929)

Related contract gap: #82 (comment)
Upstream contract issue: AgentWorkforce/relayfile#238