Skip to content

[IOTDB-1667] Speed up the deletion of database by avoiding flush#12445

Merged
OneSizeFitsQuorum merged 3 commits intomasterfrom
drop_database_avoid_flush
Apr 29, 2024
Merged

[IOTDB-1667] Speed up the deletion of database by avoiding flush#12445
OneSizeFitsQuorum merged 3 commits intomasterfrom
drop_database_avoid_flush

Conversation

@HTHou
Copy link
Copy Markdown
Contributor

@HTHou HTHou commented Apr 29, 2024

Description

https://issues.apache.org/jira/browse/IOTDB-1667

How to test

Add a UT in DataRegionTest.
Before this PR, syncDeleteDataFiles costs 101ms ~ 1121ms.
After this PR, syncDeleteDataFiles costs about 12ms.

  @Test
  public void testIoTDBTabletWriteAndDeleteDataRegion()
      throws QueryProcessException, IllegalPathException, WriteProcessException,
          TsFileProcessorException {
    String[] measurements = new String[2];
    measurements[0] = "s0";
    measurements[1] = "s1";
    TSDataType[] dataTypes = new TSDataType[2];
    dataTypes[0] = TSDataType.INT32;
    dataTypes[1] = TSDataType.INT64;

    MeasurementSchema[] measurementSchemas = new MeasurementSchema[2];
    measurementSchemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
    measurementSchemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);

    long[] times = new long[100];
    Object[] columns = new Object[2];
    columns[0] = new int[100];
    columns[1] = new long[100];

    for (int r = 0; r < 100; r++) {
      times[r] = r;
      ((int[]) columns[0])[r] = 1;
      ((long[]) columns[1])[r] = 1;
    }

    InsertTabletNode insertTabletNode1 =
        new InsertTabletNode(
            new QueryId("test_write").genPlanNodeId(),
            new PartialPath("root.vehicle.d0"),
            false,
            measurements,
            dataTypes,
            measurementSchemas,
            times,
            null,
            columns,
            times.length);

    dataRegion.insertTablet(insertTabletNode1);

    for (int r = 50; r < 149; r++) {
      times[r - 50] = r;
      ((int[]) columns[0])[r - 50] = 1;
      ((long[]) columns[1])[r - 50] = 1;
    }

    InsertTabletNode insertTabletNode2 =
        new InsertTabletNode(
            new QueryId("test_write").genPlanNodeId(),
            new PartialPath("root.vehicle.d0"),
            false,
            measurements,
            dataTypes,
            measurementSchemas,
            times,
            null,
            columns,
            times.length);

    dataRegion.insertTablet(insertTabletNode2);
    long start = System.nanoTime();
    dataRegion.syncDeleteDataFiles();
    System.out.println(System.nanoTime() - start);
    Assert.assertEquals(0, SystemInfo.getInstance().getTotalMemTableSize());

    QueryDataSource queryDataSource =
        dataRegion.query(
            Collections.singletonList(new PartialPath(deviceId, measurementId)),
            deviceId,
            context,
            null,
            null);

    Assert.assertEquals(0, queryDataSource.getSeqResources().size());
    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
  }

@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 29, 2024

Codecov Report

Attention: Patch coverage is 93.33333% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 42.93%. Comparing base (85f47e0) to head (4779801).
Report is 3 commits behind head on master.

Files Patch % Lines
.../iotdb/db/storageengine/dataregion/DataRegion.java 75.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #12445      +/-   ##
============================================
- Coverage     43.37%   42.93%   -0.45%     
  Complexity       69       69              
============================================
  Files          3102     3102              
  Lines        191553   191660     +107     
  Branches      22806    22835      +29     
============================================
- Hits          83085    82280     -805     
- Misses       108468   109380     +912     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

@OneSizeFitsQuorum OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@OneSizeFitsQuorum OneSizeFitsQuorum merged commit 9ea3695 into master Apr 29, 2024
@OneSizeFitsQuorum OneSizeFitsQuorum deleted the drop_database_avoid_flush branch April 29, 2024 10:08
HTHou added a commit that referenced this pull request Jun 26, 2024
)

* Avoid flush when drop database

* Add more UT

* recover UT
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
…che#12445)

* Avoid flush when drop database

* Add more UT

* recover UT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants