2.6.0版本，binlogrecycle动作开始后，在无外部链接情况下，内存持续增长

[wanjianjun123]

问题描述：
使用memtier_benchmark进行tendis 2.6.0版本压测，100% set，持续压测半小时，压测结束后（此时tendis无任何外部请求），当tendis触发binlogrecycle动作时，tendis进程占用的内存持续增长，最大达到202G。（再次配置binlog-save-logs = 0，测试后，内存未持续增长，怀疑是否与binlogrecycle的机制有关？）

测试环境：
tendis：2.6.0
linux：CentOS Linux release 7.7.1908 (Core)
memtier_benchmark：memtier_benchmark 1.3.0

压测命令：
memtier_benchmark -t 20 -c 50 -s xx.xx.xx.xx -p 3901 -a 123456 --distinct-client-seed --command="set key data" --key-prefix="kv_" --key-minimum=1 --key-maximum=50000000 --random-data --data-size=128 --test-time=1800

tendis配置文件：

pidfile /data/tendis/test-tendis1/tmp/tendisplus.pid
logdir /data/tendis/test-tendis1/log
dir /data/tendis/test-tendis1/db
dumpdir /data/tendis/test-tendis1/dump
slowlog /data/tendis/test-tendis1/slowlog/slowlog

requirepass 123456
masterauth 123456

daemon no
loglevel notice
cluster-enabled 0
storage rocks
maxclients 200000
jeprof-auto-dump 0
checkkeytypeforsetcmd no

slowlog-file-enabled 1
slowlog-log-slower-than 100000

binlog-enabled 1 
binlog-save-logs 1
minBinlogKeepSec 3600
truncateBinlogIntervalMs 1000
truncateBinlogNum 5000000
binlogDelRange 500000

noexpire false

netiothreadnum 32
executorWorkPoolSize 8
executorthreadnum 64
incrPushThreadnum 25
fullPushThreadnum 10
fullReceiveThreadnum 5
logRecycleThreadnum 5

binlogRateLimitMB 64

kvstorecount 50
rocks.blockcachemb 153600
rocks.blockcache_strict_capacity_limit 0
rocks.cache_index_and_filter_blocks 0
rocks.max_open_files -1

rocks.flush_log_at_trx_commit 0
rocks.compress_type lz4
rocks.level_compaction_dynamic_level_bytes 1
rocks.target_file_size_base 67108864
rocks.max_bytes_for_level_base 536870912
rocks.write_buffer_size 67108864
rocks.max_write_buffer_number 64
rocks.max_write_buffer_number_to_maintain 2
rocks.min_write_buffer_number_to_merge 2
rocks.max_background_jobs 64
rocks.max_background_compactions 60
rocks.max_background_flushes 4
rocks.max_subcompactions 4
rocks.level0_file_num_compaction_trigger 8
rocks.level0_slowdown_writes_trigger 48
rocks.level0_stop_writes_trigger 64

rocks.db_write_buffer_size 107374182400
rocks.rowcachemb 25600
rocks.use_direct_reads 1
rocks.report_bg_io_stats 0

rocks.allow_concurrent_memtable_write 1
rocks.enable_write_thread_adaptive_yield 1

rocks.enable_blob_files 1
rocks.min_blob_size 2097152
rocks.blob_file_size 67108864
rocks.blob_compression_type lz4 
rocks.enable_blob_garbage_collection 1
rocks.blob_garbage_collection_age_cutoff 0.25

cluster-migration-slots-num-per-task 100
cluster-migration-rate-limit 1000
cluster-migration-binlog-iters 20
cluster-migration-distance 10000
cluster-migration-timeout 10

cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-require-full-coverage yes

cluster-slave-no-failover no

mapping-command dbsize emptyint
mapping-command keys emptymultibulk
rename-command flushdb flushdb_rename
rename-command flushall flushall_rename
rename-command flushalldisk flushalldisk_rename

[wanjianjun123]

我理解的binlogrecycle是将rocksdb中的binlog回收掉，binlog-save-logs是控制是否将binlog落盘到dump目录，理论上这2个动作都不应该会占用大量内存且不自动释放，不知道是不是我的理解有问题，辛苦大佬解答~~谢谢！

[raffertyyu]

http://tendis.cn/#/Tendisplus/%E7%9F%A5%E8%AF%86%E5%BA%93/FAQ?id=q-tendis%e7%9a%84%e5%86%85%e5%ad%98%e5%8d%a0%e7%94%a8%e6%98%af%e5%a4%9a%e5%b0%91
可以参考一下官网上的文档

[wanjianjun123]

http://tendis.cn/#/Tendisplus/%E7%9F%A5%E8%AF%86%E5%BA%93/FAQ?id=q-tendis%e7%9a%84%e5%86%85%e5%ad%98%e5%8d%a0%e7%94%a8%e6%98%af%e5%a4%9a%e5%b0%91 可以参考一下官网上的文档

您好，文档我看过的，内存上涨确实没有超过设定的最大值，我不太理解的点是：binlogrecycle期间，持续占用内存，这个逻辑是否合理呢？此时是否会影响正常的读写性能呢？binlogrecycle不是个清理动作嘛，binlog落盘需要使用内存，那落完盘为啥不把内存释放掉呢？

[takenliu]

rocks.blockcachemb 153600
因为这里配置了153GB的blockcache，binlog回收的时候会读binlog，就会把数据写入blockcache缓存起来。

[wanjianjun123]

rocks.blockcachemb 153600 因为这里配置了153GB的blockcache，binlog回收的时候会读binlog，就会把数据写入blockcache缓存起来。

是的，我的疑问是，当binlog回收完，为什么不把占用的cache释放掉呢？感觉这个不太合理。

[takenliu]

释放不了，这是rocksdb的blockcache，读取的数据会加入到blockcache中，然后一直存在，直到有新的数据加入到blockcache里面来。你的场景下没有读请求，在有读请求的场景下，会有新的热数据加入到blockcache中来。