Skip to content

Extract context patterns from webhook schemata - #745

Merged
woodruffw merged 45 commits into
mainfrom
ww/schema-contexts
May 22, 2025
Merged

woodruffw merged 45 commits into
mainfrom
ww/schema-contexts

Conversation

@woodruffw

@woodruffw woodruffw commented May 4, 2025 •

Copy link
Copy Markdown
Member

WIP. Still thinking through how I want to approach this.

Fixes #674.

Signed-off-by: William Woodruff william@yossarian.net

@woodruffw woodruffw added the enhancement New feature or request label May 4, 2025
@woodruffw woodruffw added this to the 1.8.0 milestone May 9, 2025
@woodruffw

woodruffw commented May 13, 2025 •

Copy link
Copy Markdown
Member Author

Noting for myself: doing a linear scan over thousands of fixed contexts for each dynamic context (i.e. parsed from expressions) is going to get slow, quick. This is probably going to require a prefix tree/trie structure so that I can do efficient prefix lookups.

Edit: Candidate: https://docs.rs/trie-rs/latest/trie_rs/#modules

OTOH a much simpler approach here is a regex/DFA transformation.

Related for me to look at: DAFSAs, DAWGs, etc.

https://en.wikipedia.org/wiki/Deterministic_acyclic_finite_state_automaton

woodruffw added 2 commits May 15, 2025 23:11
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
woodruffw added a commit that referenced this pull request May 16, 2025
Broken out of #745.

Signed-off-by: William Woodruff <william@yossarian.net>
woodruffw added 2 commits May 15, 2025 23:36
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
@woodruffw woodruffw changed the title [WIP] experiment with extracting contexts from webhook schemata Extract context patterns from webhook schemata May 16, 2025
@woodruffw
woodruffw marked this pull request as ready for review May 16, 2025 03:55
@woodruffw woodruffw modified the milestones: 1.8.0, 1.9.0 May 20, 2025
woodruffw added 8 commits May 20, 2025 16:04
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
@woodruffw

Copy link
Copy Markdown
Member Author

Preliminary benchmarking shows a mixed effect on performance: running with --no-online-audits is about 9% slower on python/cpython, but about 6% faster on grafana/grafana.

(This is primarily a correctness change so even a small performance hit would be worth it, but it's good to see that it actually makes things faster in some cases.)

woodruffw added 5 commits May 22, 2025 14:50
This essentially double-parses every expression for
a check that's almost but not quite equivalent to
the dataflow contexts collection.

Removing it gives us a small performance boost
and makes the intent clearer.

Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
@woodruffw
woodruffw merged commit 46f6157 into main May 22, 2025
@woodruffw
woodruffw deleted the ww/schema-contexts branch May 22, 2025 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Handle template-injection allowlisting schematically

1 participant