Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistent connection in the keystore #623

Merged
merged 1 commit into from Aug 30, 2019

Conversation

@aphelionz
Copy link
Contributor

aphelionz commented May 9, 2019

Overview

This PR implements the orbit-db-storage-adapter paradigm which has two main benefits:

  1. Greater performance (see benchmarks below)
  2. Easier Interop to redis, mongo, sqlite at the cache and keystore level via the storage adapter

Benchmarks

Before:

$ node benchmarks/benchmark-add.js
Starting IPFS daemon...
95 queries per second, 95 queries in 1 seconds (Oplog: 95)
87 queries per second, 182 queries in 2 seconds (Oplog: 182)
80 queries per second, 262 queries in 3 seconds (Oplog: 262)
88 queries per second, 350 queries in 4 seconds (Oplog: 350)
87 queries per second, 437 queries in 5 seconds (Oplog: 437)
85 queries per second, 522 queries in 6 seconds (Oplog: 522)
83 queries per second, 605 queries in 7 seconds (Oplog: 606)
88 queries per second, 693 queries in 8 seconds (Oplog: 693)
81 queries per second, 774 queries in 9 seconds (Oplog: 774)
--> Average of 86.1 q/s in the last 10 seconds

After:

$ node benchmarks/benchmark-add.js                             
Starting IPFS daemon...                                        
451 queries per second, 451 queries in 1 seconds (Oplog: 451)  
490 queries per second, 941 queries in 2 seconds (Oplog: 941)  
494 queries per second, 1435 queries in 3 seconds (Oplog: 1435)
513 queries per second, 1948 queries in 4 seconds (Oplog: 1948)
440 queries per second, 2388 queries in 5 seconds (Oplog: 2388)
415 queries per second, 2803 queries in 6 seconds (Oplog: 2803)
416 queries per second, 3219 queries in 7 seconds (Oplog: 3219)
375 queries per second, 3594 queries in 8 seconds (Oplog: 3594)
432 queries per second, 4026 queries in 9 seconds (Oplog: 4026)
--> Average of 445.3 q/s in the last 10 seconds  

Depends On

This PR depends on the following PRs:

TODO

  • Replication still fails intermittently due to underlying js-ipfs bug (still function)
  • Deal with failing v0 tests
  • Deal with skipped tests
  • Update docs
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
src/orbit-db-address.js Outdated Show resolved Hide resolved
src/OrbitDB.js Show resolved Hide resolved
@aphelionz

This comment has been minimized.

Copy link
Contributor Author

aphelionz commented Jun 4, 2019

@shamb0t @haadcode All tests are passing except one replication test, and it's similar (though not the same) to the issues we're seeing in the v0 tests in ipfs-log. We're really close though.

I'm a bit spent today so I'm gonna take a rest but tomorrow I feel like we can together get this taken care of.

Note that I had to remove the usage of MemStore from test/replicate.test.js because it hadn't been updated to the base32 hashes. We can fix that up and restore it, or remove it completely.

package.json Show resolved Hide resolved
src/OrbitDB.js Show resolved Hide resolved
src/OrbitDB.js Outdated Show resolved Hide resolved
@aphelionz aphelionz force-pushed the fix/persistent-keystore branch 3 times, most recently from 7f228f0 to 967d717 Jul 8, 2019
test/counterdb.test.js Outdated Show resolved Hide resolved
@aphelionz aphelionz force-pushed the fix/persistent-keystore branch 3 times, most recently from dce33ba to 26c5f47 Aug 29, 2019
src/OrbitDB.js Outdated Show resolved Hide resolved
@aphelionz aphelionz force-pushed the fix/persistent-keystore branch from 2a2f653 to e24a5a6 Aug 30, 2019
pointing to fix/store-performance branch

fix:typo

fix: indentation

test: Changing test to reflect new orbit-store default

Update package.json

test: updating tests

update localstorage-level-migration dep

experiment:Moving keystore up ALL way

orbitdb storage adapter mark 1

fix: more passing tests

more fixes

chore:package-lock.json

reverting mkdir.c for now

package-lock.json for node 10.13

fix: circleci

fix: webpack fs updates

disabling loadCache

Moving storage adapter to its own thing

tests: fixing up

chore: long needed fixing

More linting

tests: fix up look sharp

test: v0 failure only

Reversting lint fixes

fix v0-load test

set cache heads

fix: passing in storage no longer needed

fix: removing artifact from previous merge

fix: honor default keystore and pesky call-by-reference bug

fix: removing directory arg from _addManifestToCache

chore: package-lock

fix: pending drop test

removing directory option for individual dbs

docs: removing directory options

fix: removing line instead of commenting

fix: moving storage setup to createInstance

feat: Upgrading ipfs to 0.36

chore: package-log

fix: restoring onlyHash

workaround: removing memstore from replication tests

fix: this.keystore.close and this.cache.close

chore: removing eslint annotation

chore: package-lock.json

fix: passing preCreate in as option

chore: package files

Fixing package.json

fixing replicate tests

Fixing some tests

Updating orbit-db-store dependency

CircleCI updates - To be obviated via other PR

Restoring ability to pass a custom directory to orbitdb.create

More test fixes

set identity tests fixed

Fixing replication tests

Temporarily disabling concurrency tests

Closing keystore in identities test

Restoring test:all

package.json

More replicate test fixes

successful make rebuild

Linting fixes
@aphelionz aphelionz force-pushed the fix/persistent-keystore branch from 5c233b5 to 47a18b0 Aug 30, 2019
@aphelionz aphelionz merged commit dcbce5e into master Aug 30, 2019
1 check passed
1 check passed
ci/circleci Your tests passed on CircleCI!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.