Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: VirtualFree of 0 bytes failed with errno=487 (ERROR_INVALID_ADDRESS) #35465

Closed
heschik opened this issue Nov 8, 2019 · 10 comments
Assignees
Milestone

Comments

@heschik
Copy link
Contributor

@heschik heschik commented Nov 8, 2019

Funny looking trybot failure on 32-bit Windows: https://storage.googleapis.com/go-build-log/0c8f6cc0/windows-386-2008_c0fd3e00.log

cc @mknyszek maybe?

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 8, 2019

The 0 bytes thing is a red herring, the code tries to decommit successively smaller chunks because of how windows mapping works.

It's almost certainly related to my code somehow, and maybe related to #35445.

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 8, 2019

On second thought... probably not related to #35445. The page size on 386 is 4 KiB.

@mknyszek mknyszek self-assigned this Nov 8, 2019
@bcmills bcmills added this to the Go1.14 milestone Nov 8, 2019
@heschik

This comment has been minimized.

Copy link
Contributor Author

@heschik heschik commented Nov 12, 2019

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 12, 2019

@heschik have you been seeing this often? Do you have more examples? I've been unable to reproduce thus far, and also I don't see it happening at all in CI which leads me to believe it's fairly rare.

I have one suspicion: the background scavenger is scavenging unreserved memory. The evidence here is that:

  • The failure is ERROR_INVALID_ADDRESS (though MSDN won't tell me under what conditions different error codes are set).
  • The page to be scavenged is at the very top of a palloc chunk, which would be the first thing the scavenger sees.

There is a check that guards against this though, so if this is indeed the case, that must be erroneously passing. But if that check is erroneously passing, then it's possible to allocate into unmapped memory, which is not a failure mode I've seen yet (though certainly possible).

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 12, 2019

Ah, excellent. Thank you @heschik.

This supports my hypothesis, though it's curious that it's the same test every time. It's probably just quite good at tickling some edge case.

I was able to reproduce once in a gomote (unfortunately it dumped the output into my terminal window and I lost all the useful info) but it's timing out now. Now that I can reproduce though it should be easier to get to the bottom of this.

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 13, 2019

I think I found the problem. Will push a fix in a bit.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 13, 2019

Change https://golang.org/cl/206978 mentions this issue: runtime: check summary before scavenging in fast path

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 13, 2019

@heschik confirmed this fixes the problem in the x/tools test (10 consecutive runs and no failure; the tests in that package take a while to run). It's a bit wild to me how consistently that test was able to trigger this, but it was nice to have a consistent reproducer. :)

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 14, 2019

@gopherbot gopherbot closed this in d183253 Nov 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.