Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 #38827

Closed
BorisKozo opened this issue May 3, 2020 · 7 comments

Comments

@BorisKozo
Copy link

@BorisKozo BorisKozo commented May 3, 2020

Note that I read carefully through #37927 and #35112 but it seems those issues were handled already.

What version of Go are you using (go version)?

Upgrade 1.13.10 -> 1.14.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Windows 10 (reproduces on other Windows versions as well)

What did you do?

Run my application memory benchmark

What did you expect to see?

Same or better results as in 1.13.10

What did you see instead?

Worse results

I have to add that our application is resource critical so every increase in memory or CPU usage raises eyebrows immediately.

I compiled our application with both GO 1.13.10 and 1.14.2 and the CPU and memory behavior are inconsistent between the two versions as the later consumes more memory (enough to trigger the benchmark failure). The behavior of the memory release is also different (see charts below) as there is no visible de-allocation of memory in 1.14.2.
I also tried changing the GOGC to various values but even at 10 the memory did not reach the behavior of 1.13.10 and CPU usage was considerable

go_13_GC_100

Initial run compiled with GO 1.13.10

Same code was compiled and run on the same machine with GOGC values as follows:
go_14_GC_10

GOGC = 10

go_14_GC_50

GOGC = 50

go_14_GC_100

GOGC = 100

go_14_GC_200

GOGC = 200

go_14_GC_500

GOGC = 500

in Issue #37525 @mknyszek mentions that another issue will be opened but I could not find the new issue.

Thanks.

@BorisKozo BorisKozo changed the title Increase in memory usage and behavior upgrading from 1.13.10 to 1.14.2 Increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 May 3, 2020
@agnivade
Copy link
Contributor

@agnivade agnivade commented May 3, 2020

Would you be able to share the output of setting GODEBUG=gctrace=1,gcpacertrace=1 in 1.13.10 vs 1.14.2 ? That would help us a lot in the investigation. Thanks.

@aclements @mknyszek

@agnivade agnivade changed the title Increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 May 3, 2020
@BorisKozo
Copy link
Author

@BorisKozo BorisKozo commented May 3, 2020

Hi, I generated the traces:
1.13.10
trace_1_13_10.txt

and
1.14.2
trace_1_14_2.txt

Look at the chart for 1.14.2, doesn't appear to be releasing anything and I am running with GOGC=100

2020-05-03_1554

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented May 4, 2020

Regarding memory release, #38617 brought our attention to a bug on Windows where in Go 1.14 the background scavenger wasn't running at all (the synchronous/eager scavenger and debug.FreeOSMemory should still work though). It's not clear to me yet that this is exactly the same problem, but try patching Go with https://golang.org/cl/229997 and see if it helps. Based on the traces you provided, I could see it helping.

Even with that patch, you may not be seeing the same log as in Go 1.13.10 with GODEBUG=gctrace=1 because we moved the scavenge trace under its own flag, which prints once per GC cycle. The flag for Go 1.14+ is GODEBUG=scavtrace=1.

@networkimprov
Copy link

@networkimprov networkimprov commented May 4, 2020

@gopherbot add OS-Windows

@gopherbot gopherbot added the OS-Windows label May 4, 2020
@BorisKozo
Copy link
Author

@BorisKozo BorisKozo commented May 4, 2020

@mknyszek I compiled master at a1ffbe9
and it seems the problem has been resolved.
go_14_fix

Thank you very much!
Which release is this fix planned to?

@BorisKozo BorisKozo closed this May 4, 2020
@networkimprov
Copy link

@networkimprov networkimprov commented May 4, 2020

I posted a backport request. If there's no intervening security update, it will land in 1.14.3.

EDIT: #38856

@BorisKozo
Copy link
Author

@BorisKozo BorisKozo commented May 4, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.