Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Aho-Corasick for alts of literals during regexp AST compilation #221

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Use Aho-Corasick for alts of literals during regexp AST compilation #221

wants to merge 23 commits into from

Commits on Aug 5, 2020

  1. Pass in an existing fsm for ast_compile().

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    510c688 View commit details
    Browse the repository at this point in the history

  2. Pass in an existing fsm for trie_to_fsm().

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    b732ae2 View commit details
    Browse the repository at this point in the history

  3. Pass in an existing fsm for re_comp() and re_strings().

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    9a58888 View commit details
    Browse the repository at this point in the history

  4. Regenerated for re_comp().

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    07f8a67 View commit details
    Browse the repository at this point in the history

  5. Catch-all for when -DNDEBUG is absent.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    a1924a4 View commit details
    Browse the repository at this point in the history

  6. Clarification.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    58a5443 View commit details
    Browse the repository at this point in the history

  7. No need for re_strings_build_new(); we can construct an fsm inside … 

    …`re_strings()`, which is already a convenience.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    97c9cde View commit details
    Browse the repository at this point in the history

  8. Missing free.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    31c7d57 View commit details
    Browse the repository at this point in the history

  9. Oops... the start state here should have been exposed when passing in… 

    … an fsm.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    fd67a02 View commit details
    Browse the repository at this point in the history

  10. Whitespace.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    469f77d View commit details
    Browse the repository at this point in the history

  11. A bugfix; fsm_unionxy() may modify the start state due to `fsm_merg… 

    …e()`.

This was neglected for the previous commit (of passing in an fsm for `ast_compile()`). Here I am attempting to rectify that by passing along the start states explicitly.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    4023754 View commit details
    Browse the repository at this point in the history

  12. An interface to add a trie entry by walking an array of AST nodes. 

    This avoids needing to construct an intermediate string, and allocating storage for it.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    bb9124c View commit details
    Browse the repository at this point in the history

  13. No need to set an end state when the caller already does this.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    83c3eaa View commit details
    Browse the repository at this point in the history

  14. Add re_strings_build_into(), to pass in a pre-existing end state.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    5e027ef View commit details
    Browse the repository at this point in the history

  15. Handling for accepting states which are not leaf nodes in the trie. 

    When a single end state is provided, we cannot set these as accepting. So here I'm hooking them up with epsilons, instead. Yes, this means we don't always produce a DFA.

To do this, we need to identify leaf nodes, so I've introduced `has_child()` for that.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    b418b9f View commit details
    Browse the repository at this point in the history

  16. A first attempt at using Aho Corasick for suitable alt nodes.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    96c1055 View commit details
    Browse the repository at this point in the history

  17. Typo.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    b74e7d8 View commit details
    Browse the repository at this point in the history

  18. Perhaps sensible thresholds?

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    0666f88 View commit details
    Browse the repository at this point in the history

  19. Sprinkle const.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    0d380c9 View commit details
    Browse the repository at this point in the history

  20. Move out ast_compile_altlist().

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    fc8a967 View commit details
    Browse the repository at this point in the history

  21. Special case for using A-C for an anchored list of alts at the root n… 

    …ode.

This allows for accepting states in the middle of branches in the trie, rather than hooking them up using epsilons.
    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    076e0ca View commit details
    Browse the repository at this point in the history

  22. Recurr for special top-level handling of group nodes.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    2d03655 View commit details
    Browse the repository at this point in the history

  23. Some tests for top-level Aho Corasick behaviour.

    @katef
    katef committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    69523b2 View commit details
    Browse the repository at this point in the history