INDY-1205: Use RocksDB as a key-value storage #561

sergey-shilov · 2018-03-06T13:30:22Z

No description provided.

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

ghost · 2018-03-06T13:30:24Z

Could one of the committers please verify this patch?

dsurnin · 2018-03-06T13:34:51Z

plenum/config.py

@@ -59,7 +60,7 @@
 clientBootStrategy = ClientBootStrategy.PoolTxn

 hashStore = {
-    "type": HS_FILE
+    "type": HS_LEVELDB


could you please use rocksdb here

No, thanks)

Seriously speaking, I've made temporary roll back to leveldb as rocksdb building procedure is not ready yet.

ashcherbakov · 2018-03-06T13:47:40Z

plenum/persistence/rocksdb_hash_store.py

+logger = getlogger()
+
+
+class RocksDbHashStore(HashStore):


The implementation looks exactly the same as LevelDbHashStore. Maybe use just one implementation with Dependency Injection of a proper key-value implementation?

Yes, but I propose to merge hash stores implementations after rocksdb build procedure is ready.

ashcherbakov · 2018-03-06T13:49:27Z

plenum/server/node.py

@@ -940,7 +932,7 @@ def onStopping(self):

    def closeAllKVStores(self):
        # Clear leveldb lock files
-        logger.debug("{} closing level dbs".format(self), extra={"cli": False})
+        logger.debug("{} closing level/rocks dbs".format(self), extra={"cli": False})


A better comment: closing key-value storages

ashcherbakov · 2018-03-06T13:50:27Z

plenum/test/storage/test_rocksdb_hash_store.py

+
+
+@pytest.yield_fixture(scope="module")
+def rocksdbHashStore(tdir):


Are there the same tests as for LevelDbHashStore? maybe just use the same test with parametrized fixtures?

ashcherbakov · 2018-03-06T13:53:25Z

storage/kv_store_rocksdb.py


    def close(self):
-        raise NotImplementedError
+        del self._db


Why do we call del here?

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

…ksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

…torage. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

sergey-shilov · 2018-03-07T13:01:46Z

Test this please.

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

sergey-shilov · 2018-03-12T07:53:29Z

Test this please.

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

…deb from sovrin. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

…bIntKeys. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

ashcherbakov · 2018-03-20T15:48:44Z

storage/kv_store_rocksdb_int_keys.py

+        opts.comparator = IntegerComparator()
+        self._db = rocksdb.DB(self._db_path, opts)
+
+    def get_equal_or_prev(self, key):


Why don't we implement it in KeyValueStorageRocksdb?

Because it is correct just in case of integer keys with corresponding comparator. Otherwise the order is lexicographical where '2' is greater than '100', so get_equal_or_prev() does not make sense.

* INDY-1205: Add rocksdb as a key-value backend. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add tests for rocksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add rocksdb support. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Use rocksdb as a backend storage. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fix initialisation of rocksdb kvstore. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fix test_kv_rocksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fix comparator of KeyValueStorageRocksdbIntKeys class. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fix test_state_rocksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add unified config-based creation of hash store. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Change default hash storage from file to rocksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Integrate rocksdb into state tests. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Merge kv storages tests into single module. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Temporary rollback to leveldb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Re-factor tests. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Implement the first version of installation of rocksdb and python-rocksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Merge leveldb and rocksdb hash storages implementations into single storage. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Use RocksDB as a key-value storage. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Tempoprary use leveldb as a default storage for the ledger. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Adopt getAllTxn() for working with rocksdb iterator. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fix db_path property for leveldb and rocksdb, fix test. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add build procedure for python-rocksdb and setuptool, use librocksdb deb from sovrin. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add missed libs to docker file. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Change rocksdb package. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Change rocksdb package for 3d parties build. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Implement get_equal_or_prev() functionality for KeyValueStorageRocksdbIntKeys. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add a helper for init of k/v storage with int keys. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Add rocksdb tests for the equal-or-prev functionality. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com> * Fallback to leveldb as we do not want to migrate to rocksdb right now. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Sergey Shilov added 13 commits March 5, 2018 18:41

INDY-1205: Add rocksdb as a key-value backend.

2fc293c

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add tests for rocksdb.

904751b

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add rocksdb support.

6a694a8

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Use rocksdb as a backend storage.

747de27

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Fix initialisation of rocksdb kvstore.

fd127f6

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Fix test_kv_rocksdb.

7f9b666

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Fix comparator of KeyValueStorageRocksdbIntKeys class.

186f773

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Fix test_state_rocksdb.

c9e7a19

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add unified config-based creation of hash store.

b2359fa

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Change default hash storage from file to rocksdb.

4271498

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Integrate rocksdb into state tests.

8b6399f

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Merge kv storages tests into single module.

6ed63c3

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Temporary rollback to leveldb.

ebed50f

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

dsurnin suggested changes Mar 6, 2018

View reviewed changes

ashcherbakov reviewed Mar 6, 2018

View reviewed changes

Re-factor tests.

3e88988

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

sergey-shilov force-pushed the feature/INDY-1205 branch 3 times, most recently from 7bfc05e to f45ab67 Compare March 7, 2018 10:01

Implement the first version of installation of rocksdb and python-roc…

a551c09

…ksdb. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

sergey-shilov force-pushed the feature/INDY-1205 branch from f45ab67 to a551c09 Compare March 7, 2018 11:05

Sergey Shilov added 3 commits March 7, 2018 15:05

Merge leveldb and rocksdb hash storages implementations into single s…

268398b

…torage. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Use RocksDB as a key-value storage.

4945aba

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

b7319ef

ashcherbakov previously approved these changes Mar 7, 2018

View reviewed changes

Tempoprary use leveldb as a default storage for the ledger.

b2190a8

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

sergey-shilov dismissed ashcherbakov’s stale review via b2190a8 March 7, 2018 13:57

Adopt getAllTxn() for working with rocksdb iterator.

677d669

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Sergey Shilov added 11 commits March 12, 2018 12:57

Fix db_path property for leveldb and rocksdb, fix test.

814bf04

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

b45fe58

Add build procedure for python-rocksdb and setuptool, use librocksdb …

237ce3b

…deb from sovrin. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add missed libs to docker file.

1b84c91

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Merge remote-tracking branch 'base/master' into feature/INDY-1205

6f63c64

Change rocksdb package.

2cd538e

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Change rocksdb package for 3d parties build.

d9f34a1

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Implement get_equal_or_prev() functionality for KeyValueStorageRocksd…

39ae3d8

…bIntKeys. Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add a helper for init of k/v storage with int keys.

b976a51

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Add rocksdb tests for the equal-or-prev functionality.

3308804

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

Fallback to leveldb as we do not want to migrate to rocksdb right now.

4a2469a

Signed-off-by: Sergey Shilov <sergey.shilov@dsr-company.com>

ashcherbakov reviewed Mar 20, 2018

View reviewed changes

ashcherbakov approved these changes Mar 20, 2018

View reviewed changes

ashcherbakov merged commit b656455 into hyperledger:master Mar 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INDY-1205: Use RocksDB as a key-value storage #561

INDY-1205: Use RocksDB as a key-value storage #561

sergey-shilov commented Mar 6, 2018

ghost commented Mar 6, 2018

dsurnin Mar 6, 2018

sergey-shilov Mar 6, 2018

sergey-shilov Mar 6, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov Mar 7, 2018

ashcherbakov Mar 6, 2018

sergey-shilov commented Mar 7, 2018

sergey-shilov commented Mar 12, 2018

ashcherbakov Mar 20, 2018

sergey-shilov Mar 20, 2018



		@pytest.yield_fixture(scope="module")
		def rocksdbHashStore(tdir):

		logger = getlogger()


		class RocksDbHashStore(HashStore):

INDY-1205: Use RocksDB as a key-value storage #561

INDY-1205: Use RocksDB as a key-value storage #561

Conversation

sergey-shilov commented Mar 6, 2018

ghost commented Mar 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergey-shilov commented Mar 7, 2018

sergey-shilov commented Mar 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment