Skip to content

Speed up recover stage when DataNode restarting#11421

Merged
xingtanzjr merged 6 commits intomasterfrom
speed_up_restart_recover
Oct 31, 2023
Merged

Speed up recover stage when DataNode restarting#11421
xingtanzjr merged 6 commits intomasterfrom
speed_up_restart_recover

Conversation

@xingtanzjr
Copy link
Contributor

Description

  1. Make the compaction selector work only after all DataRegions are ready.
  2. Flush all memtables into disk when DataNode shutting down so that it can skip the wal redo when restarting next time
  3. fix the bug that current implementation of syncCloseAllTsFileProcessor does not ensure the checkpoint of memtable is wrote into wal checkpoint file, which may lead to redundant redo of wal entries when restarting

@sonarqubecloud
Copy link

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell B 5 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@codecov-commenter
Copy link

codecov-commenter commented Oct 30, 2023

Codecov Report

Merging #11421 (999d38b) into master (e45c251) will increase coverage by 0.01%.
Report is 7 commits behind head on master.
The diff coverage is 62.06%.

@@             Coverage Diff              @@
##             master   #11421      +/-   ##
============================================
+ Coverage     48.57%   48.58%   +0.01%     
- Complexity    24440    24484      +44     
============================================
  Files          2817     2817              
  Lines        173018   173314     +296     
  Branches      20663    20705      +42     
============================================
+ Hits          84050    84213     +163     
- Misses        88968    89101     +133     
Files Coverage Δ
...b/storageengine/dataregion/flush/FlushManager.java 70.68% <100.00%> (ø)
.../iotdb/db/storageengine/dataregion/DataRegion.java 49.85% <85.71%> (-0.43%) ⬇️
...ageengine/dataregion/memtable/TsFileProcessor.java 59.87% <60.00%> (-0.18%) ⬇️
...g/apache/iotdb/db/storageengine/StorageEngine.java 31.29% <25.00%> (-0.06%) ⬇️
...org/apache/iotdb/db/service/IoTDBShutdownHook.java 0.00% <0.00%> (ø)

... and 46 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xingtanzjr xingtanzjr merged commit 1dd5a20 into master Oct 31, 2023
@HTHou HTHou deleted the speed_up_restart_recover branch February 2, 2024 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants