[feat] - Replace regexp pkg w/ go-re2 in detectors

[ahrav]

Description:

Replace the standard library regexp with go-re2, a drop-in regexp library that benchmarks significantly faster for our typical detector input size (100 bytes - 10kB). go-re2 shows close to 10x performance gains on large test data, which matches the ~10kB chunk size our detectors operate on. By swapping the regexp library, we stand to see major efficiency improvements in our detectors without changing any logic. go-re2 is optimized for input in the 100 byte to 10kB range, making it a natural fit for our ~10kB chunk data. This change will enable faster and more scalable detectors as our input grows over time.

Note: The go-re2 authors state the library performs worse than standard regexp on tiny (~10 byte) input data, as shown in benchmarks. However, given our typical chunk size of ~10kB, we expect go-re2 to provide better performance despite this tradeoff. Our input rarely drops to 10 byte sizes, so we are making an informed decision to optimize for mid-sized data at the small cost of edge case efficiency.

Note: We cannot use a go mod replace directive since that is not allowed on standard library packages. The regexp import path is thus updated for all detectors. ref thread in which Russ explains why this is not possible.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[zricethezav]

Amazing! This looks like the best regex replacement lib out there:

go-re2 is a drop-in replacement for the standard library regexp package which uses the C++ re2 library for improved performance with large inputs or complex expressions. By default, re2 is packaged as a WebAssembly module and accessed with the pure Go runtime, wazero. This means that it is compatible with any Go application, regardless of availability of cgo.

very cool

[trufflesteeeve]

It's potentially not relevant, but I wanted to check with y'all on how we handle non-utf-8 strings currently. Mostly because of this quote from go-re2:

Invalid utf-8 strings are treated differently. The standard library silently replaces invalid utf-8 with the unicode replacement character. This library will stop consuming strings when encountering invalid utf-8.

[dustin-decker]

It's potentially not relevant, but I wanted to check with y'all on how we handle non-utf-8 strings currently. Mostly because of this quote from go-re2:

Invalid utf-8 strings are treated differently. The standard library silently replaces invalid utf-8 with the unicode replacement character. This library will stop consuming strings when encountering invalid utf-8.

Great catch. Sounds like something we need to address first.

[ahrav]

It's potentially not relevant, but I wanted to check with y'all on how we handle non-utf-8 strings currently. Mostly because of this quote from go-re2:

Invalid utf-8 strings are treated differently. The standard library silently replaces invalid utf-8 with the unicode replacement character. This library will stop consuming strings when encountering invalid utf-8.

I added some tests in the huggingface_test.go file to test with invalid utf-8 sequences. So I think we might be okay? Actually I think we are fine because we convert our []byte to a string here and Go handles replacing the invalid utf-8 characters for us, thus the regex library will be operating on valid utf-8 sequences.

[rgmz]

I would test the base64 decoder as well. It seems to produce invalid (?) unicode sequences if the entire input isn't valid base64.