feat(ingestion): add event filter evaluation step to ingestion pipeline#52651
Draft
pl wants to merge 1 commit intopl/ingestion/event_filters/migrationfrom
Draft
feat(ingestion): add event filter evaluation step to ingestion pipeline#52651pl wants to merge 1 commit intopl/ingestion/event_filters/migrationfrom
pl wants to merge 1 commit intopl/ingestion/event_filters/migrationfrom
Conversation
69c5cda to
75dccf9
Compare
eli-r-ph
reviewed
Mar 28, 2026
| return true | ||
| case 'not': | ||
| return treeHasConditions(node.child) | ||
| case 'and': |
Contributor
There was a problem hiding this comment.
🤔 lines 44-45 (not condition) return !treeHasConditions(node.child) ?
Contributor
There was a problem hiding this comment.
oh nm this is the condition check, got it 👍
eli-r-ph
approved these changes
Mar 28, 2026
Contributor
eli-r-ph
left a comment
There was a problem hiding this comment.
Still in draft, happy to loop back if you add more but this LGTM 👍 can prestamp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
We need the Node.js ingestion pipeline to evaluate customer-configured event filters and drop (or count in dry run) matching events before they reach transformations.
Stacked on #52649 (migration PR).
Changes
Adds event filter evaluation to the Node.js ingestion pipeline:
common/event-filters/module:schema.ts— zod schemas for filter tree nodes and DB rows, withEventFilterMode(disabled/dry_run/live)evaluate.ts— recursive tree evaluation with conservative empty-group handling (empty AND = false, not vacuous true)manager.ts—EventFilterManagerusingBackgroundRefresher(60s) to load active filters from Postgrescommon/steps/apply-event-filters-step.ts— pipeline step in post-team preprocessing:droppedapp metricwould_be_droppedapp metricclickhouse_app_metrics2viaAPP_METRICS_OUTPUTWiring:
EventFilterManagercreated iningestion-consumer.ts, passed throughjoined-ingestion-pipeline→post-team-preprocessing-subpipelineHow did you test this code?
pnpm test -- event-filters): schema validation, tree evaluation (empty groups, conditions, AND/OR/NOT, complex trees), manager behavior (valid/invalid rows, dry_run/live modes, empty trees)bin/test-event-filtersscript on local dev environmentPublish to changelog?
No