HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location #6113

xichen01 · 2024-01-28T12:29:31Z

What changes were proposed in this pull request?

Preventing V3 Schema from Creating Container DB in the Wrong Location.

Currently, when the HddsVolume cannot load the "V3 Schema Container DB", it will create a "V3 Schema Container DB" in the logged-in user's home directory (dbOptions.setCreateIfMissing(true);). If there are multiple abnormal HddsVolume, all the Containers from these abnormal Volumes will using the same DB.
For HddsVolume that loads the Container DB abnormally, we should just make it a Failed Volume.

root cause

When the HddsVolume load "V3 Schema Container DB" abnormally, the containerData.getVolume().getDbParentDir() will be null.
The

new File(containerData.getVolume().getDbParentDir(), OzoneConsts.CONTAINER_DB_NAME);

will be

new File(null, OzoneConsts.CONTAINER_DB_NAME);

then

new File(null, OzoneConsts.CONTAINER_DB_NAME).getAbsolutePath()

will return ${user.dir}/container.db

java.io.File#getAbsolutePath:

  /**
      ...
    * If this abstract pathname is the empty abstract pathname then
    * the pathname string of the current user directory, 
      ...
  **/
    public String getAbsolutePath() {
        return fs.resolve(this);
    }

KeyValueContainerLocationUtil#getContainerDBFile:

  public static File getContainerDBFile(KeyValueContainerData containerData) {
    if (containerData.hasSchema(OzoneConsts.SCHEMA_V3)) {
      return new File(containerData.getVolume().getDbParentDir(),
          OzoneConsts.CONTAINER_DB_NAME);
    }
    return getContainerDBFile(containerData.getMetadataPath(), containerData);
  }

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10230

How was this patch tested?

Unit Test

…ng Location

adoroszlai · 2024-01-28T12:41:01Z

Thanks @xichen01 for working on this. There are some test failures:

https://github.com/xichen01/ozone/actions/runs/7685544604/job/20943567917#step:6:11

(TestOzoneContainerWithTLS also failed for me locally.)

ChenSammi · 2024-01-29T07:44:23Z

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

    if (result != VolumeCheckResult.HEALTHY ||
        !df.getContainerSchemaV3Enabled() || !isDbLoaded()) {
      return result;
    }

    // Check that per-volume RocksDB is present.
-    File dbFile = new File(dbParentDir, CONTAINER_DB_NAME);
-    if (!dbFile.exists() || !dbFile.canRead()) {
+    File dbFile = dbParentDir == null ? null : new File(dbParentDir, CONTAINER_DB_NAME);


If it passes L261 check, does it still have the chance to have a null dbParentDir?

dbParentDir will non-null if it passes L261 check. I think this can be simplified

ChenSammi · 2024-01-29T07:53:31Z

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java

    } catch (IOException e) {
-      throw new IOException("Can't init db instance under path "
-          + containerDBPath + " for volume " + getStorageID(), e);
+      dbLoadFailure.set(true);


@xichen01 , thanks for reporting this. Can you share more detail about which operation will cause the DB load failure and throw the IOException, besides the known initPerDiskDBStore? Is there any exception stack can be shared?

Not sure of the root cause of the DB loading issue, but we found a very large container.db in the system home directory, as well as error logs from HddsVolume initing.

logs from HddsVolume initing

container.db in the system home

Manually destroy the RocksDB file. 2. Then restart DN and write data. can reproduce a similar problem.

ERROR org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer: Load db store for HddsVolume /xxx/xxx/xxx/ozone/hdds failed java.io.IOException: Can't init db instance under path /xxx/xxxx/xxx/ozone/hdds/CID-xxx/DS-xxx/container.db for volume DS-xxx at org.apache.hadoop.ozone.container.common.volume.HddsVolume.loadDbStore(HddsVolume.java:235) at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.loadAllHddsVolumeDbStore(HddsVolumeUtil.java:99) at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.<init>(OzoneContainer.java:146) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.<init>(DatanodeStateMachine.java:153) at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:295) at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:227) at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:195) at org.apache.hadoop.ozone.HddsDatanodeService.call(HddsDatanodeService.java:104) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172) at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550) at picocli.CommandLine.parseWithHandler(CommandLine.java:2485) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:96) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87) at org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:178) Caused by: java.io.IOException: Failed init RocksDB, db path : /xxx/xxx/xxx/ozone/hdds/CID-xxx/DS-xxx/container.db, exception :org.rocksdb.RocksDBException CURRENT file does not end with newline at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:130) at org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:191) at org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.start(AbstractDatanodeStore.java:146) at org.apache.hadoop.ozone.container.metadata.AbstractDatanodeStore.<init>(AbstractDatanodeStore.java:99) at org.apache.hadoop.ozone.container.metadata.DatanodeStoreSchemaThreeImpl.<init>(DatanodeStoreSchemaThreeImpl.java:54) at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.getUncachedDatanodeStore(BlockUtils.java:84) at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.initPerDiskDBStore(HddsVolumeUtil.java:79) at org.apache.hadoop.ozone.container.common.volume.HddsVolume.loadDbStore(HddsVolume.java:232) ... 18 more

I think we need to root cause this issue. Could you kindly check the related DN logs to see if there is any related suspicious logs? If we don't have the root cause, then this extra IOException catch may not enough.

Not found others related log which can help to locate the root cause, this may be a legacy issue that may have existed when the cluster was created, but with this PR, we could have found this issue faster.

I have observed another impact when volume metadata path is removed incorrectly in test env. But at that point of time, container was getting loaded having NPE in other flow.
NPE was handled with #5921 avoiding loading of container.
Since it was test script issue where cleanup was not proper, referring similar case.

@xichen01 , in case we don't know the exact case of DB loading failure, let's change this catch from IOException to Throwable.

@xichen01 , could you please address my above comment and provide a new patch?

…ce some unnecessary changes

ChenSammi · 2024-04-03T06:33:13Z

...gration-test/src/test/java/org/apache/hadoop/ozone/container/server/TestContainerServer.java

@@ -192,6 +195,9 @@ private static HddsDispatcher createDispatcher(DatanodeDetails dd, UUID scmId,
    conf.set(OZONE_METADATA_DIRS, TEST_DIR);
    VolumeSet volumeSet = new MutableVolumeSet(dd.getUuidString(), conf, null,
        StorageVolume.VolumeType.DATA_VOLUME, null);
+    Path tempDir = Files.createTempDirectory("");


This tempDir should be explicitly deleted, other wise it will leak disk space on CI server, resulting out of space error at last.
Same for change in TestSecureContainerServer.java.

ChenSammi · 2024-04-08T07:32:23Z

Thanks @xichen01 for the contribution.

…ong Location (apache#6113)

…ong Location (apache#6113) (cherry picked from commit 6ed1e58)

xichen01 added 2 commits January 28, 2024 20:02

HDDS-10230 Preventing V3 Schema from Creating Container DB in the Wro…

1a3ea17

…ng Location

reduce unused change

f42c4f4

ChenSammi reviewed Jan 29, 2024

View reviewed changes

xichen01 changed the title ~~HDDS-10230 Preventing V3 Schema from Creating Container DB in the Wrong Location~~ HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location Jan 30, 2024

xichen01 added 7 commits March 27, 2024 17:07

Catch the Throwable Exception that occur when loading exception; Redu…

148cd24

…ce some unnecessary changes

Fix test

fade60c

Merge branch 'master' into HDDS-10230

0344edf

Fix test

9b344dd

findbugs

a736e71

Fix test

564c710

fix test

30c978b

ChenSammi reviewed Apr 3, 2024

View reviewed changes

xichen01 added 3 commits April 3, 2024 23:08

Replace uncleaned temporary directory

32cd826

Merge branch 'master' into HDDS-10230

92fe762

Replace uncleaned temporary directory

2205d39

ChenSammi approved these changes Apr 8, 2024

View reviewed changes

ChenSammi merged commit 6ed1e58 into apache:master Apr 8, 2024
39 checks passed

Tejaskriya pushed a commit to Tejaskriya/ozone that referenced this pull request Apr 17, 2024

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wr…

dad3600

…ong Location (apache#6113)

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 29, 2024

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wr…

9874ee9

…ong Location (apache#6113) (cherry picked from commit 6ed1e58)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location #6113

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location #6113

xichen01 commented Jan 28, 2024 •

edited

Loading

adoroszlai commented Jan 28, 2024

ChenSammi Jan 29, 2024

xichen01 Mar 27, 2024

ChenSammi Jan 29, 2024

xichen01 Jan 29, 2024

xichen01 Jan 29, 2024

xichen01 Jan 29, 2024

ChenSammi Jan 30, 2024

xichen01 Jan 30, 2024

sumitagrawl Jan 30, 2024

ChenSammi Feb 28, 2024

ChenSammi Mar 27, 2024

xichen01 Mar 27, 2024

ChenSammi Apr 3, 2024

ChenSammi commented Apr 8, 2024

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location #6113

HDDS-10230. Preventing V3 Schema from Creating Container DB in the Wrong Location #6113

Conversation

xichen01 commented Jan 28, 2024 • edited Loading

What changes were proposed in this pull request?

root cause

What is the link to the Apache JIRA

How was this patch tested?

adoroszlai commented Jan 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChenSammi commented Apr 8, 2024

xichen01 commented Jan 28, 2024 •

edited

Loading