Description
A user reports that they are seeing a potential memory leak which appears to be easily reproducible and needs further analysis and correction.
Reproduction steps from reporter:
- set GOMEMLIMIT=40MiB
- Invoke with --scrape.interval=5s
- Give it a decent toml file with some queries that return some data (not sure how necessary this is really, we just run it with ~10 or 15 queries that return a moderate amount of data)
- Watch mem usage in mem tool of choice
- I've captured the go stats that the exporter itself collects, and the only one that I could see grow (that wasn't a total metric) was go_memstats_buck_hash_sys_bytes but not sure if that's the problem.
- Ran pprof with inuse_object and inuse_bytes repeatedly over a couple of days and see plenty of GC and what not, but can't see anything not getting GC'd or "dangling" but that may just be my inexperience with pprof.
Reporter commented that the same issue appears to occur with upstream iamseth code (with different db library/driver, etc.).
Reporter commented that they found very little on the buck_hash and the best they could find is that it has to do with internal representations of maps in Go. Said this somewhat jives with what they're seeing in pprof in that none of the "application code" shows up as suspect of object counts ever-increasing or even heap size increasing over and over. That made them wonder if there was some validity to the buck bytes stuff in that maybe that's not seen in pprof
Seems to be reproducible on linux x86_64 and macOS M2 (and in a container on linux)
Reporter provided this script that they use to perform reproduction/test runs:
export LD_LIBRARY_PATH="/path/to/instantclient_19_8:$LD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export QUERY_TIMEOUT='15'
export DB_USERNAME='c##odbe'
export DB_PASSWORD=''
export DB_CONNECT_STRING='localhost:1521/FREE'
os="$(uname -s | tr [A-Z] [a-z])"
#arch="$(uname -m)"
# manually set arch since it will return arm64 on m2
arch="amd64"
rm -Rf dist
make go-build-$os-$arch
if [ $? -eq 0 ]; then
export GOMAXPROCS=1
export GOGC=75
export GOMEMLIMIT=40MiB
./dist/oracledb_exporter-2.0.0.$os-$arch/oracledb_exporter --scrape.interval=5s
fi