driver: bad connection

[redelang]

After upgrading from 1.3.1 to 1.5.2 the error 'driver: bad connection' occurs randomly. On random moments and random metrics (both for default metrics as provided custom metrics) as it seems.

The setup in which it is used:

• RHEL 8 virtual machine
• Using env variables for connection (DB_CONNECT_STRING, DB_USERNAME, DB_PASSWORD)
• Using Oracle instant client 19.14.0 and 19.25.0 (provided by LD_LIBRARY_PATH)
• Target database is PDB version 19.22.0.0

I have not made any changes in configuration when upgrading.

[markxnelson]

Thanks for reporting - we have been seeing this too, and are investigating already. Have not identified root cause yet.

[markxnelson]

Just to confirm - you are running the binary right, not running the exporter in a container?

[redelang]

Yes indeed, running the binary.

[markxnelson]

I have a suspicion it might be in godror layer. we call into that (using go sql) and that calls into our oracle odpi-c layer then that into instant client's static libs. i've seen this issue with 19, 21 and 23, and amd and arm, in container and not. might be to do with conflicts between go's connection pooling and instant client's. working on testing/debugging.

have yhou noticed any issues with queries just returning no results at all? sometimes i see a query that works fine in sqlcl/sqlplus but always has no results when run through this same stack of deps. usually more complex ones. would be good to know about that too if you see that. and if you do - if you can share a query, that'd be great. i can give email address if you don't want to post it

[redelang]

Didn't really notice queries returning no results yet, but I didn't ran it for long yet. Have been focussing on the errors and put the upgrade on hold now because of those errors.

[markxnelson]

Ok thanks - and these look pretty random right, does not seem to be any particular pattern to them?

[redelang]

I didn't discover a pattern, different queries both from the default and custom provided metrics ran into this error.

[markxnelson]

Ok, thanks a lot for the info, it helps a lot, will take that into account while trying to get to the bottom of that issue.

[redelang]

btw this issue: #168 was found because of this issue. I was surprised that for some metrics there was no error in the log even though it was exposed in the scrape errors metric that these failed. It appears to be caused by the setting ignorezeroresult.

[anders-swanson]

FYI #168 is fixed by #173. It may take us some time to get to the bottom of this issue :)

[markxnelson]

We believe we have this fixed in 1.5.4 - if you still see the error, please reopen, or open a new issue to let you know. Thanks very much for your input.