HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. #981

bharatviswa504 · 2019-06-17T19:10:14Z

No description provided.

hadoop-yetus · 2019-06-17T21:32:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1002	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	90	Maven dependency ordering for branch
+1	mvninstall	609	trunk passed
+1	compile	291	trunk passed
+1	checkstyle	79	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	850	branch has no errors when building and testing our client artifacts.
+1	javadoc	164	trunk passed
0	spotbugs	360	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	561	trunk passed
		_ Patch Compile Tests _
0	mvndep	31	Maven dependency ordering for patch
+1	mvninstall	507	the patch passed
+1	compile	304	the patch passed
+1	javac	304	the patch passed
-0	checkstyle	38	hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	665	patch has no errors when building and testing our client artifacts.
+1	javadoc	163	the patch passed
+1	findbugs	587	the patch passed
		_ Other Tests _
-1	unit	109	hadoop-hdds in the patch failed.
-1	unit	2171	hadoop-ozone in the patch failed.
+1	asflicense	46	The patch does not generate ASF License warnings.
		8482

Reason	Tests
Failed junit tests	hadoop.utils.db.TestDBStoreBuilder
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestContainerStateMachine
	hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules
	hadoop.ozone.ozShell.TestOzoneShell
	hadoop.ozone.TestSecureOzoneCluster

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/Dockerfile
GITHUB PR	#981
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux e53471f1b77a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3d020e9`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/diff-checkstyle-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/testReport/
Max. process+thread count	4043 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-981/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

anuengineer

I am concerned that this is a feature that might be just over-engineering. We have always claimed that OM or SCM needs SSDs. If we have SSDs I don't think we will be bottlenecked by the IOPS. I know that RocksDB has lots of features, not all we need to expose in ozone. Also, RocksDB settings are isolated today in a class called RockDB config. It really think we should not bleed them into Ozone.

anuengineer · 2019-06-17T22:15:08Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java

@@ -35,6 +35,7 @@
  // metadata dir but in future we may support multiple for redundancy or
  // performance.
  public static final String OZONE_SCM_DB_DIRS = "ozone.scm.db.dirs";
+  public static final String OZONE_SCM_DB_WAL_DIR = "ozone.scm.db.wal.dir";



Do we know that WAL performance is being compromised today ? This adds a new config value, but are we sure about the benefit ? In other words, are we sure that ozone is being bottlenecked by RocksDB?

anuengineer · 2019-06-17T22:16:00Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java

@@ -220,4 +227,15 @@ private File getDBFile() throws IOException {
    return Paths.get(dbPath.toString(), dbname).toFile();
  }

+  private File getWALFile() throws IOException {
+    if (walPath == null) {
+      LOG.error("Write-ahead-log path is " +


Ok, in most cases this would not happen. No one would map a WAL to a different disk. If it is mapped, then we can probably log that we detected it is mapped. This is certainly not an error.

anuengineer · 2019-06-17T22:16:35Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

@@ -646,6 +646,17 @@
      production environments.
    </description>
  </property>
+  <property>


Can we please use the new config style?

https://cwiki.apache.org/confluence/display/HADOOP/Java-based+configuration+API

anuengineer · 2019-06-17T22:16:49Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

@@ -699,6 +710,17 @@
      production environments.
    </description>
  </property>
+  <property>
+    <name>ozone.scm.db.wal.dir</name>


same comment as above.

anuengineer · 2019-06-17T22:16:58Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

@@ -2327,6 +2349,17 @@
      production environments.
    </description>
  </property>
+  <property>
+    <name>ozone.recon.db.wal.dir</name>


anuengineer · 2019-06-17T22:25:56Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java

@@ -35,6 +35,7 @@
  // metadata dir but in future we may support multiple for redundancy or
  // performance.
  public static final String OZONE_SCM_DB_DIRS = "ozone.scm.db.dirs";
+  public static final String OZONE_SCM_DB_WAL_DIR = "ozone.scm.db.wal.dir";


Do we have data that indicates the compaction is the root cause of issues? On a normal SSD, we have around 80K IOPS are we saying that we are need more bandwidth than that ?

anuengineer · 2019-06-17T22:26:36Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java

@@ -220,4 +227,15 @@ private File getDBFile() throws IOException {
    return Paths.get(dbPath.toString(), dbname).toFile();
  }

+  private File getWALFile() throws IOException {
+    if (walPath == null) {
+      LOG.error("Write-ahead-log path is " +


I would think that most people will never even add WAL as an option, so this should not be an error IMHO.

anuengineer · 2019-06-17T22:27:01Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

@@ -646,6 +646,17 @@
      production environments.
    </description>
  </property>
+  <property>
+    <name>ozone.om.db.wal.dir</name>


Use new style Config options please.

anuengineer · 2019-06-17T22:28:22Z

hadoop-hdds/common/src/test/java/org/apache/hadoop/utils/db/TestDBStoreBuilder.java

+      Assert.assertTrue(newFolder.mkdirs());
+    }
+    DBStoreBuilder.newBuilder(conf)
+        .setPath(newFolder.toPath())


This test illustrates my concern. We will add this feature for performance reasons but we really have no data to show or prove this is the root cause of perf issues. Neither can we test it.

anuengineer · 2019-06-17T22:28:49Z

hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java

+            // If wal Dir is set, as in OM HA setup all OM's will be on the
+            // same node, wal directories will conflict with each other.
+            // Append nodeId similar to metaDirPath.
+            String walDir = conf.get(OMConfigKeys.OZONE_OM_DB_WAL_DIR);


New Configs please.

bharatviswa504 · 2019-06-24T23:31:37Z

Closing this, as there is a way to pass a .ini file and read the values and set the required rocksdb.
Thank You @anuengineer and @arp7 for the info.

…s thread-safe, enabling StreamRegexMonitors only when required StreamPartitionCountMOnitor and StreamRegexMonitor use the KafkaSystemAdmin, which has a kafka-consumer. Kafka consumer is not thread-safe, therefore explicit synchronization is required for metadata accesses. StreamRegexMonitor is not to be enabled when there is no regex input. Author: Ray Matharu <rmatharu@linkedin.com> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#981 from rmatharu/bugfix-regexmonitor

HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB.

d7f1881

bharatviswa504 added the ozone label Jun 17, 2019

bharatviswa504 requested a review from mukul1987 June 17, 2019 19:10

bharatviswa504 self-assigned this Jun 17, 2019

anuengineer requested changes Jun 17, 2019

View reviewed changes

bharatviswa504 closed this Jun 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. #981

HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. #981

bharatviswa504 commented Jun 17, 2019

hadoop-yetus commented Jun 17, 2019

anuengineer left a comment

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

anuengineer Jun 17, 2019

bharatviswa504 commented Jun 24, 2019

HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. #981

HDDS-1696. RocksDB use separate Write-ahead-log location for OM RocksDB. #981

Conversation

bharatviswa504 commented Jun 17, 2019

hadoop-yetus commented Jun 17, 2019

anuengineer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharatviswa504 commented Jun 24, 2019