New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MyRocks memory grows unlimitedly after a lot of compactions #1269
Comments
Rocksdb LOG with many TTL compactions:
|
You can disable TTL compactions by setting relevant RocksDB column family options (ttl and periodic_compaction_seconds) to 0. In MyRocks, you can set by rocksdb_default_cf_options='ttl=0;periodic_compaction_seconds=0' in my.cnf. Also you may need to tune (reduce) compaction unit size. For example, set max_compaction_bytes=400m through rocksdb_default_cf_options. |
RocksDB engine status:
|
Thanks for your reply. We have already disabled TTL and periodic compactions. But this problem should happen again when there are many compaction, we want to know why compaction make memory grow unlimitedly. Is there memory leak or untracked memory usage that can't be controled by configuration? |
Here is our MyRocks configuration:
|
We use tcmalloc as the allocator by setting the 'malloc-lib' option in the my.cnf.
|
It's unlikely a leak, but more likely because compaction unit size was too high (too many files were involved in a compaction at a time). You can see from LOGs about how many files were involved for compactions (grep by "Compacting "). Check OPTIONS* file (under $datadir/.rocksdb/) and see the value of max_compaction_bytes, which defines the compaction size. If you set low enough (e.g. 400mb), it will not trigger one giant compaction but trigger multiple small compactions so mem usage will be more limited. Regarding memory allocator, we recommend jemalloc, since most RocksDB developments and tests were done with jemalloc. |
It should be our fault not to set max_compaction_bytes and it is target_file_size_base * 25 by default (800M in our configuration). tcmalloc and jemalloc seem be both great and we don't know whether choosing jemalloc or tcmalloc makes different. We will try jemalloc later. In our observation of MyRocks memory usage, some memory used in compaction will never be released. Neither memtable or block cache owns these memory. This should be a problem because the doc of Rocksdb memory usage doesn't mention it. |
Try jemalloc, disable ttl compaction, and set lower max_compaction_bytes and if you still see problems on large compactions. You can trigger a whole column family compaction by "set global rocksdb_compact_cf='default';" so you can try on your test systems. |
After replacing tcmalloc with jemalloc, loading data into block cache can also make the memory usage growing problem. We configure the block cache to 95G and fill them with data. Finally, the memory usage of myrocks grows up to 120G, which is the lxc soft limit of our debian instance (using cgroup). There is no compaction when loading the data. However, when we reduce the block cache size to 10G, we can't see the problem when loading data into the block cache. Is the fragmentation of allocation that cause the problem? What is the size of block cache that we should set in myrocks? |
Here is the memory stats:
memory usage:
jeprof:
|
Do you mean LOAD DATA INFILE or large bulk inserts (GBs of data) by one transaction? |
No, just using read operation of |
@yoshinorim
|
MySQL version: percona-server 5.7.39-42
RocksDB version: 6.21.1
Recently, we found a MyRocks instance got memory growing a lot during a few hours (from 95G to 120G). Total memory usage is exceed the size of memtable and block cache (block cache: 78G, memtable: 512M * 4 ).
Then we tried to found out the reason and saw that there were many TTL compactions happens at that time. Our MyRocks did less compactions before. There is TTL compactions in RocksDB and it will compact all SST files older than 30d by default. Our MyRocks has run for 30d before the momery grows and we sync much data on its setup. That is the reason why there were a lot compactions at that time.
It seems that it is the compaction make MyRocks memory grow without control. I have observed that the memory grows a little after each compaction before, although the block cache and memtable usage have reach the limit we configure.
Why does MyRocks memory grow and how can we handle this problem? It seems that it will lead to MyRocks OOM.
Instance background: Our MyRocks is an instance for backup, we sync data to it on setup and there are a few read/write operations in its life time(30d).
The text was updated successfully, but these errors were encountered: