Skip to content

feat(agent): add keep-proxy-stream-open ingest flag#3067

Merged
charlesvien merged 1 commit into
mainfrom
feat/event-ingest-keep-proxy-stream-open
Jul 1, 2026
Merged

feat(agent): add keep-proxy-stream-open ingest flag#3067
charlesvien merged 1 commit into
mainfrom
feat/event-ingest-keep-proxy-stream-open

Conversation

@charlesvien

@charlesvien charlesvien commented Jul 1, 2026

Copy link
Copy Markdown
Member

Problem

On the event-ingest proxy path, flush() closes the active NDJSON upload after every drained batch. The ingress in front of the agent-proxy buffers the request body and only forwards it once the upload closes, so closing per batch forwards each batch within a round-trip instead of stranding events in a long-lived upload.

We want to switch the proxy path back to a single long-lived buffered upload (matching the Django path) to investigate the ingestion buffering behavior, without changing the default for every run. This puts that behavior behind the tasks-agent-proxy-keep-stream-open feature flag, off by default.

Changes

  • Add a keepProxyStreamOpen option to TaskRunEventStreamSender, sourced from POSTHOG_TASK_RUN_EVENT_INGEST_KEEP_STREAM_OPEN via AgentServerConfig.eventIngestKeepStreamOpen.
  • Gate the per-batch closeActiveStream() on the proxy write leg in flush() behind the flag. Off (default) keeps today's behavior: close per drained batch. On holds one long-lived buffered upload and delivers on the normal roll triggers and at stop.
  • The env var is set by the control plane when the tasks-agent-proxy-keep-stream-open flag is enabled (companion PR feat(tasks): add tasks-agent-proxy-keep-stream-open flag posthog#67492).
  • Cover both branches in the sender test: default closes per batch, flag on holds one upload.

How did you test this?

  • pnpm --filter @posthog/agent test event-stream-sender -> 18/18 pass.
  • pnpm --filter @posthog/agent typecheck -> clean.
  • Pre-commit hook ran repo-wide Biome + pnpm typecheck; CI green on the pushed commit.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

@charlesvien charlesvien added the Stamphog This will request an autostamp by stamphog on small changes label Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

React Doctor found no issues in the changed files. 🎉

Reviewed by React Doctor for commit 8206e88.

github-actions[bot]
github-actions Bot previously approved these changes Jul 1, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Purely additive, backward-compatible feature flag (defaults to false, preserving existing behavior) with a well-targeted test covering the new code path. No showstoppers.

@greptile-apps

greptile-apps Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "gate proxy per-batch stream close behind..." | Re-trigger Greptile

Comment thread packages/agent/src/server/event-stream-sender.test.ts Outdated
Comment thread packages/agent/src/server/event-stream-sender.test.ts Outdated
@github-actions github-actions Bot dismissed their stale review July 1, 2026 17:00

New commits pushed (delta classified non_trivial_delta) — stamphog approval dismissed; re-review running automatically.

github-actions[bot]
github-actions Bot previously approved these changes Jul 1, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple, backward-compatible feature flag (defaults false) that gates an existing per-batch stream-close call; no production risk. Bot comments flag test description and sleep-based assertion style, not correctness of the new logic or the test's core invariant.

@charlesvien charlesvien force-pushed the feat/event-ingest-keep-proxy-stream-open branch from ee106a9 to eadf9c8 Compare July 1, 2026 17:42
@github-actions github-actions Bot dismissed their stale review July 1, 2026 17:43

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

@charlesvien charlesvien force-pushed the feat/event-ingest-keep-proxy-stream-open branch from eadf9c8 to 9000935 Compare July 1, 2026 17:43
@charlesvien charlesvien changed the title feat(agent): gate proxy per-batch stream close behind flag fix(agent): don't close the proxy ingest stream per batch Jul 1, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes a deliberate production behavior — per-batch stream close on the proxy path — that was explicitly designed to work around a buffering ingress that holds request bodies until the upload closes. With no current human approval (both top-level reviews were dismissed for older commits), and no visible justification that the proxy ingress no longer buffers, this change needs a human reviewer to confirm the proxy side has been updated before merging.

@stamphog stamphog Bot removed the Stamphog This will request an autostamp by stamphog on small changes label Jul 1, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes an intentional per-batch stream-close on the proxy path that was explicitly designed to work around a buffering ingress that holds request bodies until the upload closes. No human has approved the current HEAD (both top-level reviews are dismissed for older commits), and a substantive bot comment (@stamphog) flags that this behavioral change needs a human to confirm the proxy ingress no longer buffers. Without that confirmation, removing this logic could strand events until stop() is called, risking data loss if a process exits abnormally.

@charlesvien charlesvien force-pushed the feat/event-ingest-keep-proxy-stream-open branch from 9000935 to 46df848 Compare July 1, 2026 18:03
@charlesvien charlesvien changed the title fix(agent): don't close the proxy ingest stream per batch feat(agent): add keep-proxy-stream-open ingest flag Jul 1, 2026
@charlesvien charlesvien added the Stamphog This will request an autostamp by stamphog on small changes label Jul 1, 2026
github-actions[bot]
github-actions Bot previously approved these changes Jul 1, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backward-compatible, opt-in flag defaulting to false — no existing production behavior changes without explicitly setting the env var. The bot comments about removing the per-batch close don't apply to the current implementation, which preserves that behavior by default.

@charlesvien charlesvien force-pushed the feat/event-ingest-keep-proxy-stream-open branch from 46df848 to 8206e88 Compare July 1, 2026 18:32
@github-actions github-actions Bot dismissed their stale review July 1, 2026 18:33

New commits pushed (delta classified non_linear_history) — stamphog approval dismissed; re-review running automatically.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Purely additive, backward-compatible opt-in flag defaulting to false — no existing production behavior changes without explicitly setting the env var. The stamphog bot concerns targeted an older version that removed the per-batch close; the current implementation preserves it as the default and gates the new behavior behind the flag. All inline test concerns are resolved.

@charlesvien charlesvien merged commit fd94245 into main Jul 1, 2026
23 checks passed
@charlesvien charlesvien deleted the feat/event-ingest-keep-proxy-stream-open branch July 1, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Stamphog This will request an autostamp by stamphog on small changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant