New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: possible memory corruption #49075
Comments
How often does this happen? Is there any way we could reproduce? |
@catenacyber says it is still happening multiple times a day on OSS-Fuzz. Philippe, do you have any hints about an easy way to get a reproduction case on our own machines? |
Around 50 times a day on oss-fuzz
I did not manage to reproduce it myself, did not try very hard though...
Well, the hard thing is that this bug does not reproduce for a specific input.
Then, I guess you need to wait one hour, and relaunch the fuzzer if it did not trigger the bug, until it does
Is there some environment variable to do so ? |
Maybe oss-fuzz uses |
No, you'd have to edit the code to replace pool allocations with |
so rebuild the standard library ? |
Just edit it, rebuild will be automatic. |
So, I did google/oss-fuzz#6623 with regex.go not using |
google/oss-fuzz#6623 looks worth a shot. Thanks. |
It looks like the bug is still happening but much less often. One last stack trace is
Any more clues ? |
Could you bisect to a particular commit that introduced the corruption? |
oss-fuzz uses latest Golang release, so they switched from 1.16.x to 1.17 on August 16th, but we do not know which commit exactly in this major release induced the buggy behavior... |
I have another possible theory:
Is there any way you can get at the regexp and the contents of the string? In this case, My theory is that |
Put the hex encoded version of one string here : |
If it's an alignment issue, could we try to reproduce by mmap'ing a large block and creating a string with all the different possible alignments / ending pointer bytes? |
What is the regexp that is being used? |
regexp.MustCompile( |
So, there are kinds of a lot of answers to split.... |
Nothing obvious with the string+regexp you posted. Putting it at different alignments doesn't seem to trigger anything. |
This string posted comes with stack trace
|
When running manually, it seems my string is always aligned on 16 bytes.like |
That doesn't seem right. The string posted is only 227927 bytes, but the stack trace shows that is is 9841664 bytes. To unalign a string, do:
|
So, the string must have been corrupted before the call to |
I'm not sure I understand. How did you get the string, if not just dumping the 0x962c00 bytes starting at address 0x10c000ac201f ? |
oss-fuzz provides the stack trace, and the input that triggered it. |
Hm, then I guess the values in the stack traces are incorrect. Which can happen, especially with regabi introduced in 1.17. Another thing to try, run with |
FWIW, I was not able to reproduce either using the oss-fuzz local execution steps on a clean Linux VM (amd64, Ubuntu 20.04) following the directions outlined above in #49075 (comment), with 3 separate runs that totaled about 24 hours. I also ran for about 24 hours using dvyukov/go-fuzz against the same gonids fuzz target, which also did not crash (although I don't think there is an enormous amount of signal from that result, given libFuzzer and go-fuzz have different process restart strategies, different mutations, different propensity for creating large inputs, and so on). @catenacyber some questions for you:
That said, it might be that size of inputs in the corpus is not meaningful if this turns out to be due to some corruption.
In any event, sorry if anything I wrote is off base, but curious for your thoughts. |
I did not manage to reproduce, so this may indeed be a hardware/OS issue...
I just opened google/oss-fuzz#6650 |
I am not able to reproduce myself... Just seeing crashes on oss-fuzz
You can download the oss-fuzz public corpus
I think this
Reverting the change to get a more steady flow of bugs back
Nothing was off base... |
Bug still happens for the last 10 days after merge of |
Is that setting of the env variable only happening at build time? I am not in front of a full size computer, and probably would be smarter for me to let someone else chime in, but I had thought GODEBUG=cpu.all=off was a runtime flag? |
I used it only at build time, not at runtime. |
The setting of |
Can I set the environment variable while running the program ? Or should it be set before ? (harder to set up...) |
Let's first find out whether it actually fixes the problem. |
I think what @catenacyber is asking is whether GODEBUG can be set after program start and still be effective for debugging or if it's only checked at start. |
Indeed, thanks Filippo. |
Yes, I didn't answer that question because it's complicated and depends on the exact value of |
Thanks Ian, you answered may question.
There will be no more point to do it for me. So, I guess I will think on it... |
@asraa Could these be provided in oss-fuzz logs ? |
Side note on reproducing the issue. |
No luck in reproducing it in 500 runs... |
Is there anything you can tell us about the machines that are running this task, when they fail? |
After re-reading this issue yesterday I have recalled a post by Aleksey Shipilev regarding memcheck, might be a case too (less often due to smaller amount of pointers vs jvm) https://shipilev.net/jvm/test-your-memory/ |
Possibly related to #49961 , if oss-fuzz uses |
When will it be in a release version ? |
1.17.5, probably, which would be early Jan. If you can, please try and patch in the CL (https://go-review.googlesource.com/c/go/+/369098/) and see if that fixes the issue. |
So @randall77, bug still happens even with fix for #49961 :-/ |
Is there any update on this issue? Are people still seeing the problem? Thanks. |
Still happens like 6 times a day over last week |
I am encountering a similar issue also from OSS-fuzz:
|
Thanks Adam. |
OSS-Fuzz reported an issue a few weeks ago that we suspect is memory corruption caused by the runtime. This started on August 16th, so is likely a Go 1.17 issue.
A slice bounds out of range issue is being reported from calls to
regexp.MustCompile(,\s*).Split
However, this is not reproducible with the inputs provided by OSS-Fuzz, so we expect something else is going on.
Below are some of the panic logs:
From rsc@:
The relevant code is processing the
[][]int
returned fromregexp.(*Regexp).FindAllStringIndex
.That
[][]int
is prepared by repeated append:Each of the
match[0:2]
being appended is prepared inregexp.(*Regexp).doExecute
by:appending to a zero-length, non-nil slice to copy
m.matchcap
.And each of the
m.matchcap
is associated with the*regexp.machine m
, which is kept in async.Pool
for reuse.The specific corruption is that the integers in the
[][]int
are clear non-integers (like pointers),which suggests that either one of the appends is losing the reference accidentally during GC
or something in sync.Pool is wonky.
This could also be something strange that OSS-Fuzz is doing, and doesn't necessarily represent a real-world use case.
/cc @golang/security
The text was updated successfully, but these errors were encountered: