Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperscan MPM integration v6 #1965

wants to merge 3 commits into from


Copy link

commented Mar 28, 2016

(Version 6: adds the hashlittle_safe() function in c0b7bd6 to avoid asan/valgrind warnings when hashing arbitrary strings, as discussed in v5. The previous PR was #1955.)

This PR adds support for using Intel's Hyperscan regex engine as an MPM algo, namely "hs".

It has a couple of notable features over a straightforward implementation:

  • Caseful/caseless matching is done by Hyperscan natively.
  • Offset and depth are also checked by Hyperscan using its extended parameter support (this meant that these parameters had to be passed through to the add functions in 87bd04a).
  • Hyperscan MPM structures are cached and deduped. We found during testing that some configurations generate many duplicate copies of the same MPM matcher, and this has a very significant memory cost. It might be that this can be addressed, or that deduplication can be hoisted up to apply generically to any MPM implementation.

More info on Hyperscan:

jviiret added 3 commits Mar 15, 2016
MpmAddPatternCI and MpmAddPatternCS had arguments for offset and depth,
but these were not being passed in by the caller.
This adds an MPM implementation that uses the Hyperscan regex engine
library from Intel, accessible as the "hs" mpm-algo.
By default, hashlittle() will read off the end of the key, up to the
next four-byte boundary, although the data beyond the end of the key
doesn't affect the hash. This read causes uninitialized read warnings
from Valgrind and Address Sanitizer.

Here we add hashlittle_safe(), which avoids reading off the end of the
buffer (using the code inside the VALGRIND-guarded block in the original
hashlittle() implementation).

This comment has been minimized.

@inliniac inliniac referenced this pull request Mar 29, 2016

This comment has been minimized.

Copy link

commented Mar 29, 2016

The suppressions in here make my DrMemory test pass as well: 31ed704


This comment has been minimized.

Copy link

commented Mar 30, 2016

Merged through #1968, thanks a lot Justin!

@inliniac inliniac closed this Mar 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.