[BUG]After the upgrade, the database recovery is very slow

[Benxiaohai001]

Describe the bug

After the upgrade, the database recovery is very slow.
version:
43c12d8 -> f0cf701

To Reproduce

1. start cnosdb single mode;

rm -rf /data/2.3/cnosdb
mkdir -p /data//2.3/cnosdb/logs
nohup ./target/release/cnosdb-43c12d8 run -M singleton --config ./config/config_8902.toml > /data/2.3/cnosdb/logs/data_node.1001.log &

2. insert data

create database benchmark001 with ttl '9999' shard 2;
create database benchmark002 with ttl '9999' shard 2;
create database benchmark003 with ttl '9999' shard 2;

load_cnosdb --batch-size 20000 --workers 200 --urls "http://127.0.0.1:9802/"  --db-name benchmark001 --file /data/cnosdb-iot-seed-123-scale-100-2022-01-01-one-day-10

load_cnosdb --batch-size 20000 --workers 200 --urls "http://127.0.0.1:9802/"  --db-name benchmark002 --file /data/cnosdb-iot-seed-123-scale-100-2022-01-01-one-day-10

load_cnosdb --batch-size 20000 --workers 200 --urls "http://127.0.0.1:9802/"  --db-name benchmark003 --file /data/cnosdb-iot-seed-123-scale-100-2022-01-01-one-day-10

3. kill cnosdb

kill -9 pid

4. start cnosdb process

nohup ./target/release/cnosdb-43c12d8 run -M singleton --config ./config/config_8902.toml >> /data/2.3/cnosdb/logs/data_node.1001.log &

5. repeat step 2-4 three times;
6. start cnosdb process

 nohup ./target/release/cnosdb-f0cf70160e run -M singleton --config ./config/config_8902.toml >> /data/2.3/cnosdb/logs/data_node.1001.log &

### Expected behavior

_No response_

### Additional context

1st wal recover 20min;
2nd wal recover 40 min
3 rd wal recover 40 min;
4th wal recover 40 min;
For users, this waiting time is a bit too long

[roseboy-liu]

#1175

[Benxiaohai001]

ce version 69eaec7-> 54e037f
When the wal file reaches a certain size, it will be cleaned up regularly, so as to ensure that the wal process does not take too long.The issue is verified,bug was fixed ,close;

[root@cicd_ujv23 cnosdb]# ll cnosdb/1001/wal/ -h
total 429M
-rw-r--r--. 1 root root 429M Jun 13 15:47 _000012.wal

[root@cicd_ujv23 cnosdb]# tail cnosdb/logs/data_node.1001.log  -f
2023-06-13T15:48:25.925889747+08:00  INFO tskv::kvcore: Flush task handler started
2023-06-13T15:48:25.925907039+08:00  INFO tskv::kvcore: Summary task handler started
2023-06-13T15:48:25.925988455+08:00  INFO tskv::kvcore: Job 'WAL' started.
2023-06-13T15:48:25.930366572+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/18061912559370128348", error: io error: unexpected end of file
2023-06-13T15:48:25.930482312+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/9066705053438593533", error: io error: unexpected end of file
2023-06-13T15:48:25.930545521+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/6487049532457436581", error: io error: unexpected end of file
2023-06-13T15:48:25.930607302+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/6461511650039252478", error: io error: unexpected end of file
2023-06-13T15:48:25.930677138+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/3137414998859467311", error: io error: unexpected end of file
2023-06-13T15:48:25.930726659+08:00  WARN query::dispatcher::persister: Failed to deserialize query info from file: "/root/.cnosdb/query/5130888302325285661", error: io error: unexpected end of file
blocking waiting for Ctrl-C...
2023-06-13T15:48:51.557287040+08:00  INFO meta::store::state_machine: METADATA WRITE: /data_version :101
2023-06-13T15:48:51.557421281+08:00  INFO meta::store::state_machine: METADATA WRITE: /cluster_xxx/data_nodes_metrics/1001 :{"id":1001,"disk_free":264840450048,"time":1686642531,"status":"Healthy"}
2023-06-13T15:48:51.557500877+08:00  INFO meta::store::state_machine: WRITE: /cluster_xxx/data_nodes_metrics/1001 :{"id":1001,"disk_free":264840450048,"time":1686642531,"status":"Healthy"}