Skip to content

celfmt: fold filter into consuming map during simplify - #107

Merged
efd6 merged 2 commits into
masterfrom
filter_filter
Aug 6, 2026
Merged

celfmt: fold filter into consuming map during simplify#107
efd6 merged 2 commits into
masterfrom
filter_filter

Conversation

@efd6

@efd6 efd6 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor
celfmt: fold filter into consuming map during simplify

Add a peephole optimisation that rewrites filter-then-map chains into
a single map with a filter argument:

  x.filter(v, cond).map(v, t) → x.map(v, cond, t)

@efd6 efd6 self-assigned this Jul 21, 2026
@efd6 efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Jul 21, 2026
@efd6
efd6 marked this pull request as ready for review July 21, 2026 23:52
@efd6
efd6 requested a review from a team as a code owner July 21, 2026 23:52
Add a peephole optimisation that rewrites filter-then-comprehension
chains into a single comprehension with a filter argument:

  x.filter(v, cond).map(v, t) → x.map(v, cond, t)
  x.filter(v, cond).transformList(_, v, t) → x.transformList(_, v, cond, t)
  x.filter(v, cond).transformMap(_, v, t) → x.transformMap(_, v, cond, t)
  x.filter(v, cond).transformMapEntry(_, v, t) → x.transformMapEntry(_, v, cond, t)

For two-variable comprehensions the merge is restricted to cases where
the first iteration variable (index/key) is "_", since the filter alters
element indices and merging would change semantics if the index is used.

@chrisberkhout chrisberkhout left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like reading it with the separate filter, but I guess this is okay. Is it more efficient?

I found some problems with AI that I think are real:

Bug 1: folding fails when the filter's source is itself a macro

This includes chained filters:

$ celfmt -s <<< 'state.a.filter(u, u > 0).filter(v, v < 10).map(v, v * 2)'
failed to format program: unsupported expression: 0
$ celfmt -s <<< 'state.a.map(u, u + 1).filter(v, v > 0).map(v, v * 2)'
failed to format program: unsupported expression: 0

Root cause: cel-go stores nested macro references inside a macro-call tree as unspecified exprs carrying only the ID (parser/helper.go: macroTarget = p.exprFactory.NewUnspecifiedExpr(target.ID())). The new macro call built by the fold keeps filterTarget referencing the source by its original ID, but the AST-update visitor then moves that macro entry away:

				filterSource := iterRange.AsComprehension().IterRange()
				if mcall, ok := info.GetMacroCall(filterSource.ID()); ok {
					info.SetMacroCall(iterRange.ID(), mcall)
					info.ClearMacroCall(filterSource.ID())
				}
				iterRange.SetKindCase(filterSource)

After the fold, the whole subtree is rendered from the macro tree keyed at the consuming comprehension's ID, so the source's entry must stay at its original ID — the remap breaks the reference the formatter follows. I verified the fix: deleting the if mcall, ok := ... block (keeping only iterRange.SetKindCase(filterSource)) makes both cases fold correctly (state.a.map(u, u + 1).map(v, v > 0, v * 2) etc.) and all 177 tests still pass. The remap looks like it was pattern-matched from inlineAs, where it is needed. A regression test for the chained case belongs in the table.

Bug 2: transform* folds change semantics when the source is a map

CEL's filter macro on a map iterates and yields keys, while the two-var transform* comprehensions on a map bind (key, value). So for a map source the fold rebinds the variable from filtered keys to values. Verified with mito:

state.m = {"a": 1, "b": 2, "c": 3}

original: state.m.filter(k, k != "b").transformList(_, v, v)   → ["a", "c"]
folded:   state.m.transformList(_, v, v != "b", v)             → [3, 1, 2]

The _-index guard doesn't help here — the problem isn't index reuse, it's that v now binds values instead of keys. Since celfmt compiles with state as Dyn, there's no type information to distinguish list from map sources, so I don't see a way to apply the transformList/transformMap/transformMapEntry folds soundly in general — they may need to be dropped, or gated on a syntactically provable list source. The map fold is unaffected: both filter and the map macro iterate keys on maps, so m.filter(k, c).map(v, t)m.map(v, c, t) for both source types.

Minor notes

  • The rename-capture guard (filter_map_rename_capture test) and the exact-count idempotency check are both solid.
  • After a rename, the underlying AST comprehension still uses the old variable name — only the macro tree is renamed. That's harmless today because rendering goes through the macro tree, but it leaves the AST and macro map inconsistent, same caveat as inlineAs.
  • Tests pass (177 in 2 packages), but the table has no chained-filter, macro-source, or map-source cases — exactly where both bugs live.

There's a part that is fragile for future additions:

the second switch that builds newArgs uses default: to mean "the three transform functions", and valIdx == 1 doubles as the "is a two-var comprehension" test. If someone adds a new function name to the first switch with a different shape, the second switch's default will silently index args[2] for it. Mirroring the case list in both switches (or switching on valIdx) would remove that trap.

@efd6

efd6 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Is it more efficient?

Yes, it is. The filter needs to populate a new array, so a filter in front of a map makes unnecessary allocations.

@efd6
efd6 requested a review from chrisberkhout August 5, 2026 21:01
@efd6 efd6 changed the title celfmt: fold filter into consuming comprehension during simplify celfmt: fold filter into consuming map during simplify Aug 5, 2026
@efd6
efd6 merged commit eab6112 into master Aug 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants