runtime: Go routine and writer memory not being released #32124
What version of Go are you using ( go version )?
What did you do?
What did you expect to see?
The memory usage upon leaving the scope of the function should return to the levels of memory usage before entering the function call.
What did you see instead?
Within a kubernetes environment, this application will consume a majority of the node's memory (8GB) according to Grafana. The application will continue to run at high memory levels (>2 GB) until it the finishes the function. At this point, memory usage will slowly climb down but after time it will stabilize and have released about 20-30%.
In a few tests with kubernetes, we have commented out the os.Write (tried bufio.Write too) in the writer channels. Memory usage held steadily below 200 MB, but at scale, memory being released afterwards was at most 50%.
Locally on Mac, the application memory according to the activity monitor never reached above 20MB, but the memory for cached files kept growing until it reached capacity of the macbook pro. After processing the files, the cached file memory stayed at those levels until application was terminated.
In each of tests we ran with a thousand workers(go routines) for reading and encrypt/decrypt
There is no
That sounds like a reasonable behavior for a buffered filesystem.
you are correct, the current commit I had pushed was my attempt with bufio.Write(). I was stating that we had previously used os.Write().
I'll do more tests of the memory with GOGC, but I missed stating above that we tested with debug.FreeOSMemory.
Here are some snapshots from Grafana with some of things you suggested I have tested this morning.
What I was expecting to see was that memory will be reclaimed after the channels and files are closed especially after decryption, but not how it's hanging around after 10 minutes.
On Darwin we call
With that being said in Go 1.13 we use a Darwin-specific
Also in Go 1.13, we scavenge memory much more aggressively (#30333), so you will no longer need to wait minutes for memory to start being returned to the OS.
Please try running this with Go built at tip. I think this problem should be already solved for Go 1.13.