Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
MVStore writes become unnaturally slow #1570
I'm creating and populating 32 different file databases in the same process. Each database is structured the same: default settings with 3 tables each having a primary key and a index but no foreign keys.
After about 2 hours of continuous MERGEs in batches of 10000, the database slows to a crawl. System stats show that the process is spending a ton of time reading but only a small window for writing.
Just from inspecting the stacktraces, it looks like...
This is on a modern laptop with a SSD. The speed drops from 100ms-400ms per 10000 rows to 10000ms-60000ms. Is there any reason why H2 would slow down like this?
This doesn't happen in master. The I/O rates are much more consistent, but the periodic compaction that's happening causes database access to block for a long time. At the tail end of my test, the compaction going on had my application code sitting for ~16 minutes (database was 115gb in size at this point).
Is this something that's tuneable via the JDBC URL parameters? Stack traces below...
Your stack traces look normal. It is expected that compaction of 115Gb database can take a loooong time. If your test continually makes updates, databases chankes become sparsely populated and need to be compacted. One workaround is to shutdown database at some point and have SHUTDOWN_DEFRAG=TRUE option in url, which would invoke off-line compaction on shutdown. This procedure is much more efficient, than the online compaction. Another option is to have a higher threshold for automatic compaction, but MVStore "autoCompactFillRate" configuration parameter is not available directly from url when MVStore is used within H2 database (mayby it should be). The downside of this scenario that you db file would be even bigger.