Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Unbounded SQLite instances/connections contributing to OOM failures #22650
Version Used: 15.3
Currently we fail to bound the number of instances of the following types which are created at runtime:
Associated with these types is a pair of allocations in the native heap. One is 64,000 bytes, and the other is 425,600 bytes. Ordinarily, this would not be a problem. However, it appears that it is possible for the number of connections to grow over time, resulting in overwhelming memory pressure stemming from the (mis-)use of SQLite. The following image shows one such case:
After fixing this for 15.5, we should port the fix to 15.4 servicing.
Interesting. This could definitely happen. Specifically in the case of a huge amount of concurrent calls into the persistence service.
The worst case would be:
Task-1 makes persistence call, disk is loaded and actual disk writing blocks thread. Threadpool gets full and continues spawning more tasks, which all end up doing this. Eventually disk is released and these all complete. But the pool of connections still holds onto all of these.
A reasonable cap on the pool size (maybe 16 or so) woudl avoid this. but it would not address the problem of threadpool starvation+thread-spawning that can occur if the disk is loaded.
A different sort of fix would be to have a dedicated thread for writing to the DB. Requests to write from other threads would be queued to this single thread, allowing the writes to be 'async' from the perspective of the callign thread. Slow disk would then only be an impact on that single writer.
We would also only need a single connection at that point.