Skip to content

feat(ingestion): add event filter evaluation step to ingestion pipeline#52651

Draft
pl wants to merge 1 commit intopl/ingestion/event_filters/migrationfrom
pl/ingestion/event_filters/nodejs
Draft

feat(ingestion): add event filter evaluation step to ingestion pipeline#52651
pl wants to merge 1 commit intopl/ingestion/event_filters/migrationfrom
pl/ingestion/event_filters/nodejs

Conversation

@pl
Copy link
Copy Markdown
Contributor

@pl pl commented Mar 28, 2026

Problem

We need the Node.js ingestion pipeline to evaluate customer-configured event filters and drop (or count in dry run) matching events before they reach transformations.

Stacked on #52649 (migration PR).

Changes

Adds event filter evaluation to the Node.js ingestion pipeline:

  • common/event-filters/ module:

    • schema.ts — zod schemas for filter tree nodes and DB rows, with EventFilterMode (disabled/dry_run/live)
    • evaluate.ts — recursive tree evaluation with conservative empty-group handling (empty AND = false, not vacuous true)
    • manager.tsEventFilterManager using BackgroundRefresher (60s) to load active filters from Postgres
    • Full test coverage: 46 tests across schema validation, evaluation logic, and manager behavior
  • common/steps/apply-event-filters-step.ts — pipeline step in post-team preprocessing:

    • Live mode: drops matching events, produces dropped app metric
    • Dry run mode: lets events through, produces would_be_dropped app metric
    • App metrics written to clickhouse_app_metrics2 via APP_METRICS_OUTPUT
  • Wiring: EventFilterManager created in ingestion-consumer.ts, passed through joined-ingestion-pipelinepost-team-preprocessing-subpipeline

How did you test this code?

  • 46 unit tests (pnpm test -- event-filters): schema validation, tree evaluation (empty groups, conditions, AND/OR/NOT, complex trees), manager behavior (valid/invalid rows, dry_run/live modes, empty trees)
  • Integration tested via bin/test-event-filters script on local dev environment

Publish to changelog?

No

@pl pl requested a review from a team March 28, 2026 00:50
@pl pl force-pushed the pl/ingestion/event_filters/nodejs branch from 69c5cda to 75dccf9 Compare March 28, 2026 00:54
return true
case 'not':
return treeHasConditions(node.child)
case 'and':
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 lines 44-45 (not condition) return !treeHasConditions(node.child) ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh nm this is the condition check, got it 👍

Copy link
Copy Markdown
Contributor

@eli-r-ph eli-r-ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still in draft, happy to loop back if you add more but this LGTM 👍 can prestamp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants