Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trufflehog crashes when built with Go 1.24 and run on high-end machines #3914

Closed
Snaipe opened this issue Feb 14, 2025 · 3 comments · Fixed by #3923
Closed

Trufflehog crashes when built with Go 1.24 and run on high-end machines #3914

Snaipe opened this issue Feb 14, 2025 · 3 comments · Fixed by #3923
Labels

Comments

@Snaipe
Copy link

Snaipe commented Feb 14, 2025

TruffleHog Version

Tested with these commits, compiled from source:

Note that the issue does not happen with the Github release x86_64 binary for v3.88.8. We are nevertheless impacted because we always build releases from source.

Trace Output

No trace output can be obtained with --log-level=5, trufflehog crashes in init.

Expected Behavior

Running trufflehog --version should print a version.

Actual Behavior

Any invocation of trufflehog fails with the following stack trace:

panic: wasm error: invalid table access
	wasm stack trace:
		.$27(i32,i32,i32,i32)
		.$398(i32)
		.$430(i32,i32)
		.$428(i32,i32,i32)
		.$474(i32)
		.$652(i32,i32,i32)
		.$666(i32,i32,i32) i32

goroutine 1 [running]:
github.com/wasilibs/go-re2/internal.newRE(0xc002f2c240, {0x7eb00?, 0xc001fbdc70?}, {0x5c?, 0xa?, 0x48?, 0x0?})
	/home/snaipe/go/pkg/mod/github.com/wasilibs/go-re2@v1.8.0/internal/re2_wazero.go:289 +0x269
github.com/wasilibs/go-re2/internal.Compile({0xc00007eb00, 0x35}, {0x70?, 0xdd?, 0xfb?, 0x1?})
	/home/snaipe/go/pkg/mod/github.com/wasilibs/go-re2@v1.8.0/internal/re2.go:67 +0x165
github.com/wasilibs/go-re2.Compile(...)
	/home/snaipe/go/pkg/mod/github.com/wasilibs/go-re2@v1.8.0/re2.go:46
github.com/wasilibs/go-re2.MustCompile({0xc00007eb00, 0x35})
	/home/snaipe/go/pkg/mod/github.com/wasilibs/go-re2@v1.8.0/re2.go:76 +0x2e
github.com/trufflesecurity/trufflehog/v3/pkg/detectors/abbysale.init()
	/home/snaipe/b5/src/github.com/trufflesecurity/trufflehog/pkg/detectors/abbysale/abbysale.go:29 +0x12a

Steps to Reproduce

  1. Build trufflehog with Go 1.24
  2. Run the binary on a server with a high-end CPU (tested machines all used an Intel Xeon 5318Y)

Environment

  • OS: Linux
  • Versions: 5.10.62, 5.15.106, 6.6.69

Additional Context

This bug still stumps us to this day because there's no visible side effect that might explain why this issue occurs only on certain machines. Building the trufflehog binary with Go 1.24 statically and transferring the binary on a various set of hosts, it seems that:

  1. The binary fails to run on hosts with a 96-core CPU (Intel Xeon 5318Y), while it works on an 80-core CPU (Intel Xeon 5218R) and 8-core CPU (Intel Core i7-8550U)
  2. Debugging the binary shows that on the failing host, wazero crashes in the execution engine when making a call back to Go through the Go ABI -- which likely changed with go1.24. On the successful hosts the instructions bail before said ABI call.
  3. Running the binary under strace does not reveal anything odd -- both the failing and successful runs open the same files, read the same content, and generally perform the exact same operations with the same side effects leading up to the failure point.

This failure seems to have been discovered recently in go-re2 (see wasilibs/go-re2#190), and upgrading wazero once the issue has been fixed should fix this bug.

However, given that wazero is likely to break again each time Go changes its internal ABI, and that we value far more stability across Go upgrades, we'd like to request that a build tag is made available to switch between the standard library regexp package and go-re2. The binary release can still use go-re2 by default, but users that compile their trufflehog release from source will still very likely experience this same bug when new Go versions come out.

References

@Snaipe Snaipe added the bug label Feb 14, 2025
@brent-spyder
Copy link

I am seeing this issue as well

@kashifkhan0771
Copy link
Contributor

Thanks, @Snaipe, for opening this issue! We are aware of it and will discuss internally how to prevent similar issues in the future.

@Snaipe
Copy link
Author

Snaipe commented Feb 17, 2025

At the very least, from our perspective, we don't care much about the regexp engine in use, but we prefer smoother Go upgrades in our build farm, so having some build tag to switch back to std regexp during compilation would work for us, if that option is on the table. gitleaks seems to be doing the same: https://github.com/gitleaks/gitleaks/tree/master/regexp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

3 participants