os: possible memory leak in os.ReadFile()

[g0194776]

What version of Go are you using (go version)?

1.17.2

Does this issue reproduce with the latest release?

I've tested in golang 1.18.6, it also has the same problem.

What operating system and processor architecture are you using (go env)?

[root@10 ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 4
cpu cores	: 8
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 5
cpu cores	: 8
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 6
cpu cores	: 8
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 7
cpu cores	: 8
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

[root@10 ~]# uname -a
Linux 10.202.42.22 5.2.14-1.el7.elrepo.x86_64 #1 SMP Tue Sep 10 10:50:19 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@10 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

What did you do?

My program is running in the container.

I've met a strange memory leak problem with os.ReadFile(). My program will read some binary file data by soft-link(e.g. /proc/TARGET-PID/exe). After several days, I had found that my program process RSS memory usage is increasing and the pprof heap shows that only far smaller than RSS memory is used. The biggest part of memory usage just points to os.ReadFile().

As profile003 has shown, currently the heap just has 384MB but the target process RSS is 691.45MB.

Here is the heap dump.

heap.out.zip

I also manually set GODEBUG=madvdontneed=1 to speed RSS memory collection up, but it didn't work.

What did you expect to see?

All the garbage memory can be collected.

What did you see instead?

Target process RSS is increasing, and the memory read from os.ReadFile() could not be GC.

[D1CED]

This is totally expected given that GOGC=100 is set (the default). This means that the Go runtime keeps the process heap at about double the size of all reachable objects (live heap). You can set GOGC to a lesser value but then CPU load will increase.

The Go team has recently published a really nice piece of documentation with interactive views explaining the workings of the garbage collector https://go.dev/doc/gc-guide.

[mvdan]

What @D1CED said. Also, see https://golang.org/wiki/Questions.

[g0194776]

@D1CED @mvdan
Thanks for your help. I know what you said, but apparently, it's different. That was one of my cases, In my other case, the RSS was up to 3.4GB, and the heap usage was still 500~600MB.

I would like to know how can I debug it, or what kind of evidence should I upload.

[mvdan]

Please see https://github.com/golang/go/wiki/Questions if you're looking for help to debug what's going on. You can always come back to the issue tracker when you've got a memory leak that can be reproduced.

[g0194776]

@mvdan Okay, I really need your help. I will reproduce it after the process RSS over 3GB and generate a pprof heap dump uploading here.

[Jayuda]

Do you solved this issue ?
I have this problems too. On go 1.19

[g0194776]

Not yet. @Jayuda