Skip to content

fix empty resource recovered from wal#11452

Merged
xingtanzjr merged 2 commits intomasterfrom
fix_recover_empty_resource
Nov 7, 2023
Merged

fix empty resource recovered from wal#11452
xingtanzjr merged 2 commits intomasterfrom
fix_recover_empty_resource

Conversation

@HTHou
Copy link
Contributor

@HTHou HTHou commented Nov 1, 2023

Description

#11404 fixed when inserting empty Tablet and flush, some empty TsFileResource generated. However, after inserting empty Tablet, killing and restarting DataNode, the empty TsFileResource will still be generated.

To fix it, we need to add the deletion logic of #11404 to the position where resource recovered from WAL.

How to reproduce

  1. start 1C1D iotdb and execute the following code.

  public static void main(String[] args)
      throws IoTDBConnectionException, StatementExecutionException {
    session =
        new Session.Builder()
            .host(LOCAL_HOST)
            .port(6667)
            .username("root")
            .password("root")
            .version(Version.V_1_0)
            .build();
    session.open(false);
    insertTablet();
    session.close();
  }


  private static void insertTablet() throws IoTDBConnectionException, StatementExecutionException {
    // The schema of measurements of one device
    // only measurementId and data type in MeasurementSchema take effects in Tablet
    List<MeasurementSchema> schemaList = new ArrayList<>();
    schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
    schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
    schemaList.add(new MeasurementSchema("s3", TSDataType.INT64));

    Tablet tablet = new Tablet(ROOT_SG1_D1, schemaList, 100);

    // Method 1 to add tablet data
    long timestamp = System.currentTimeMillis();

    for (long row = 0; row < 100; row++) {
      int rowIndex = tablet.rowSize++;
      tablet.addTimestamp(rowIndex, timestamp);
      for (int s = 0; s < 3; s++) {
        tablet.addValue(schemaList.get(s).getMeasurementId(), rowIndex, null);
      }
      if (tablet.rowSize == tablet.getMaxRowNumber()) {
        session.insertTablet(tablet, true);
        tablet.reset();
      }
      timestamp++;
    }

    if (tablet.rowSize != 0) {
      session.insertTablet(tablet);
      tablet.reset();
    }
  }
  1. kill datanode by kill -9 <pid>
  2. restart datanode, then print the TsFileResource. You will see the TsFileResource doesn't contains any devices.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Nov 1, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@codecov-commenter
Copy link

Codecov Report

Merging #11452 (9db0ad9) into master (e7592c0) will decrease coverage by 0.03%.
Report is 1 commits behind head on master.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #11452      +/-   ##
============================================
- Coverage     49.14%   49.12%   -0.03%     
+ Complexity    24556    24546      -10     
============================================
  Files          2768     2768              
  Lines        172197   172200       +3     
  Branches      20603    20604       +1     
============================================
- Hits          84619    84585      -34     
- Misses        87578    87615      +37     
Files Coverage Δ
.../iotdb/db/storageengine/dataregion/DataRegion.java 50.61% <0.00%> (-0.11%) ⬇️

... and 20 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@wangchao316 wangchao316 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xingtanzjr xingtanzjr merged commit 3244255 into master Nov 7, 2023
@HTHou HTHou deleted the fix_recover_empty_resource branch November 8, 2023 07:11
HTHou added a commit that referenced this pull request Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants