open rocksdb graph store using multi-threads when multi disks #721

zhoney · 2019-10-28T13:38:00Z

Change-Id: I83525ff97942c49c80f3767ed19d891617ae54f9

codecov · 2019-10-28T13:52:04Z

Codecov Report

Merging #721 into master will decrease coverage by 1.18%.
The diff coverage is 86.36%.

@@             Coverage Diff              @@
##             master     #721      +/-   ##
============================================
- Coverage     71.41%   70.22%   -1.19%     
+ Complexity     4328     4298      -30     
============================================
  Files           280      281       +1     
  Lines         20593    20737     +144     
  Branches       2893     2922      +29     
============================================
- Hits          14706    14563     -143     
- Misses         4444     4729     +285     
- Partials       1443     1445       +2

Impacted Files	Coverage Δ	Complexity Δ
...raph/backend/store/rocksdb/RocksDBStdSessions.java	`69.68% <100%> (+0.03%)`	`28 <0> (-1)`	⬇️
...ugegraph/backend/store/rocksdb/RocksDBOptions.java	`99.09% <100%> (+0.07%)`	`2 <0> (ø)`	⬇️
.../hugegraph/backend/store/rocksdb/RocksDBStore.java	`70.28% <79.31%> (+6.91%)`	`56 <4> (+11)`	⬆️
...raph/backend/store/postgresql/PostgresqlStore.java	`0% <0%> (-100%)`	`0% <0%> (-2%)`
...raph/backend/store/postgresql/PostgresqlTable.java	`0% <0%> (-100%)`	`0% <0%> (-20%)`
...kend/store/postgresql/PostgresqlStoreProvider.java	`0% <0%> (-92.11%)`	`0% <0%> (-5%)`
...aph/backend/store/postgresql/PostgresqlTables.java	`0% <0%> (-87.81%)`	`0% <0%> (-1%)`
...backend/store/postgresql/PostgresqlSerializer.java	`0% <0%> (-84.62%)`	`0% <0%> (-4%)`
...h/backend/store/postgresql/PostgresqlSessions.java	`0% <0%> (-70.28%)`	`0% <0%> (-8%)`
.../hugegraph/backend/store/AbstractBackendStore.java	`72.22% <0%> (-13.5%)`	`5% <0%> (+1%)`
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2785a09...b616401. Read the comment docs.

Change-Id: I83525ff97942c49c80f3767ed19d891617ae54f9

Linary · 2019-10-31T03:34:44Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -273,6 +300,22 @@ public void close() {

        this.checkOpened();
        this.sessions.close();
+
+        boolean terminated;


boolean terminated = false better

Change-Id: Id1f61eb808b5782a137f53c852cbb5aa48c6f870

Change-Id: Ib25b9138aed71e3b4e60e1ab9927c8813cf58f1d

javeme · 2019-11-01T11:39:54Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -69,6 +74,11 @@
    private RocksDBSessions sessions;
    private final Map<HugeType, String> tableDiskMapping;

+    private static final String DB_OPEN = "db-open-";


"db-open-%s"

javeme · 2019-11-01T11:40:20Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

+                }));
+            }
+        }
+        for (Future<?> f : futures) {


Future<?> future

javeme · 2019-11-01T12:02:00Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -273,6 +300,22 @@ public void close() {

        this.checkOpened();
        this.sessions.close();
+


ensure sessions for all threads are closed

javeme · 2019-11-01T12:05:41Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

+            try {
+                f.get();
+            } catch (Throwable e) {
+                throw new BackendException("Failed to open RocksDB store", e);


seems can destroy thread pool after open

Change-Id: Ibf6aac1a35e0eda76f07d66875e25f1d169dd638

Change-Id: Ic893d41710b51634aa9a9f962d49ba33291e4056

javeme · 2019-11-03T14:44:47Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

            } catch (Throwable e) {
                throw new BackendException("Failed to open RocksDB store", e);
            }
        }
+        this.shutdownOpenPool();


should create pool before submit for every open()

Change-Id: Iad1330c553839aec3a3238fc9f42fef13faf6463

javeme · 2019-11-04T03:37:47Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBOptions.java

+    public static final ConfigOption<Boolean> SKIP_STATS_UPDATE_ON_DB_OPEN =
+            new ConfigOption<>(
+                    "rocksdb.skip_stats_update_on_db_open",
+                    "This flag allows us to not update statistics.",


Whether to skip statistics update when opening the database,
setting this flag true allows us to not update statistics.

javeme · 2019-11-04T03:39:48Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

@@ -151,20 +159,59 @@ public synchronized void open(HugeConfig config) {
            return;
        }

+        List<Future<?>> futures = new ArrayList<>();
+        ExecutorService openPool = ExecutorUtil.newFixedThreadPool(8, DB_OPEN);


can also use MAX_FILE_OPENING_THREADS?

this thread pool is used to open DBs, not RocksDB sst files.

Change-Id: I80234139e2fa2a08d7f94bf0bde2056efb4b6ee9

zhoney force-pushed the rocksdb-speedsup branch 2 times, most recently from 9e3f63f to 276fd3d Compare October 29, 2019 08:33

open rocksdb graph store using multi-threads when multi disks

9f69d9d

Change-Id: I83525ff97942c49c80f3767ed19d891617ae54f9

zhoney force-pushed the rocksdb-speedsup branch from 276fd3d to 9f69d9d Compare October 29, 2019 09:26

Linary requested changes Oct 31, 2019

View reviewed changes

zhoney added 2 commits October 31, 2019 14:49

improve

47c349e

Change-Id: Id1f61eb808b5782a137f53c852cbb5aa48c6f870

support option.max_manifest_file_size arg

79b88d2

Change-Id: Ib25b9138aed71e3b4e60e1ab9927c8813cf58f1d

javeme requested changes Nov 1, 2019

View reviewed changes

zhoney added 2 commits November 3, 2019 13:58

improve

44b19c8

Change-Id: Ibf6aac1a35e0eda76f07d66875e25f1d169dd638

improve

ee7a0e4

Change-Id: Ic893d41710b51634aa9a9f962d49ba33291e4056

javeme requested changes Nov 3, 2019

View reviewed changes

improve

0cb2160

Change-Id: Iad1330c553839aec3a3238fc9f42fef13faf6463

javeme requested changes Nov 4, 2019

View reviewed changes

improve

b616401

Change-Id: I80234139e2fa2a08d7f94bf0bde2056efb4b6ee9

javeme approved these changes Nov 4, 2019

View reviewed changes

Linary approved these changes Nov 4, 2019

View reviewed changes

Linary merged commit 8ee971f into master Nov 4, 2019

Linary deleted the rocksdb-speedsup branch November 4, 2019 05:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

open rocksdb graph store using multi-threads when multi disks #721

open rocksdb graph store using multi-threads when multi disks #721

zhoney commented Oct 28, 2019

codecov bot commented Oct 28, 2019 •

edited

Linary Oct 31, 2019

javeme Nov 1, 2019

javeme Nov 1, 2019

javeme Nov 1, 2019

javeme Nov 1, 2019

javeme Nov 3, 2019

javeme Nov 4, 2019

javeme Nov 4, 2019

zhoney Nov 4, 2019

		@@ -273,6 +300,22 @@ public void close() {

		this.checkOpened();
		this.sessions.close();

open rocksdb graph store using multi-threads when multi disks #721

open rocksdb graph store using multi-threads when multi disks #721

Conversation

zhoney commented Oct 28, 2019

codecov bot commented Oct 28, 2019 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 28, 2019 •

edited