Potential memory leak

[markxnelson]

A user reports that they are seeing a potential memory leak which appears to be easily reproducible and needs further analysis and correction.

Reproduction steps from reporter:

• set GOMEMLIMIT=40MiB
• Invoke with --scrape.interval=5s
• Give it a decent toml file with some queries that return some data (not sure how necessary this is really, we just run it with ~10 or 15 queries that return a moderate amount of data)
• Watch mem usage in mem tool of choice
• I've captured the go stats that the exporter itself collects, and the only one that I could see grow (that wasn't a total metric) was go_memstats_buck_hash_sys_bytes but not sure if that's the problem.
• Ran pprof with inuse_object and inuse_bytes repeatedly over a couple of days and see plenty of GC and what not, but can't see anything not getting GC'd or "dangling" but that may just be my inexperience with pprof.

Reporter commented that the same issue appears to occur with upstream iamseth code (with different db library/driver, etc.).

Reporter commented that they found very little on the buck_hash and the best they could find is that it has to do with internal representations of maps in Go. Said this somewhat jives with what they're seeing in pprof in that none of the "application code" shows up as suspect of object counts ever-increasing or even heap size increasing over and over. That made them wonder if there was some validity to the buck bytes stuff in that maybe that's not seen in pprof

Seems to be reproducible on linux x86_64 and macOS M2 (and in a container on linux)

Reporter provided this script that they use to perform reproduction/test runs:

export LD_LIBRARY_PATH="/path/to/instantclient_19_8:$LD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export QUERY_TIMEOUT='15'
export DB_USERNAME='c##odbe'
export DB_PASSWORD=''
export DB_CONNECT_STRING='localhost:1521/FREE'

os="$(uname -s | tr [A-Z] [a-z])"
#arch="$(uname -m)"
# manually set arch since it will return arm64 on m2
arch="amd64"
rm -Rf dist
make go-build-$os-$arch
if [ $? -eq 0 ]; then
    export GOMAXPROCS=1
    export GOGC=75
    export GOMEMLIMIT=40MiB
    ./dist/oracledb_exporter-2.0.0.$os-$arch/oracledb_exporter --scrape.interval=5s
fi

[markxnelson]

Investigation of this issue, including several long running tests, produced the following findings:

• Go garbage collection seems to be working correctly, and there does not appear to be any memory leak in the exporter code itself (verified with pprof).
• The Go runtime seems to keep blocks of memory that are no longer needed as a result of GC, in case they may be needed again in the future. This is a documented, standard behavior. It will generally not release that memory unless there is pressure on the host. This behavior can be confirmed by looking at the process memory map and gdb.
• Go provides a GODEBUG setting called madvdontneed which allows you to adjust this behavior. Setting this to 1, and calling debug.FreeOSMemory() will cause the Go runtime to release that memory back to the OS more aggressively. Note that it is not guaranteed, but you should see RSS decrease much sooner in this scenario than the default bahavior.
• Memory usage can further be controlled with the GOMAXPROCS, GOGC and GOMEMLIMIT env vars

As a result, some new features were introduced in #55 to allow finer control of memory usage.

References and interesting articles:

• https://pkg.go.dev/runtime#hdr-Environment_Variables
• https://pkg.go.dev/runtime/debug#FreeOSMemory
• https://stackoverflow.com/questions/59599846/go-1-13-rss-keeps-on-increasing-suspected-scavenging-issue
• https://stackoverflow.com/questions/64963912/memory-is-not-even-getting-freed-back-to-os
• https://utcc.utoronto.ca/~cks/space/blog/programming/GoProgramMemoryUse
• https://forum.golangbridge.org/t/when-i-use-godebug-madvdontneed-1-and-freeosmemory-the-memory-not-being-reclaimed-by-os-for-long-time-why/32331

[markxnelson]

Fixed in #55 and 1.2.0 release