Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support db2 #49

Merged
merged 5 commits into from Feb 5, 2021
Merged

Support db2 #49

merged 5 commits into from Feb 5, 2021

Conversation

arj03
Copy link
Member

@arj03 arj03 commented Feb 2, 2021

This PR adds support for db2 by checking if db2 has been loaded in secret stack and if so uses a level index in the same way it used a flumeview-reduce index before.

In this PR I also took the liberty of doing a bit of cleaning up. I converted the CI to github actions so we have tests running again here. I moved some of the glue code into a separate folder and made it possible disable them. I don't like modules doing this kind of spooky interactions, so I would rather have ssb-friends do 1 thing and that is to give a friends graph and then leave all the glueing of the modules together in another place.

I'll squash the commits for a nicer history, but first I want to put this into ssb-browser and see how it behaves.

@arj03
Copy link
Member Author

arj03 commented Feb 3, 2021

Dictionary compression test seems to be working a bit better. Test from my browser profile seeing roughly 1000 feeds at hops 2.

total data 1148126
getting contacts state: 97.003ms
feeds: 10035
parsing contacts state: 41.602ms
json size of feeds directly 561960
json size of contacts 3978489

The last 2 numbers are only for comparison. What they mean is that the dictionary compression got the size down from roughly 4mb json to 1.1mb. Still level is wierd:

36M	/tmp/ssb-browser-sync/db2/indexes/contacts

This is not ready for review yet, just dotting down numbers as I get closer.

@arj03 arj03 mentioned this pull request Feb 3, 2021
@arj03
Copy link
Member Author

arj03 commented Feb 3, 2021

Ho ho ho, turns out level is insanely naive when it comes to batches. Massive speedup incoming :)

@arj03
Copy link
Member Author

arj03 commented Feb 3, 2021

This last commit fixes the batching which in turn have several nice benefits:

  • faster initial indexing
  • much smaller final size (784k instead of 36M)

@arj03
Copy link
Member Author

arj03 commented Feb 4, 2021

After running this in the browser for a bit of testing I discovered a bug. I think the bug is actually in ssb-db2.

@arj03
Copy link
Member Author

arj03 commented Feb 4, 2021

Found the problem. It's a bit wierd that you can shadow a const by doing a let in a different scope :(

@KyleMaas
Copy link

KyleMaas commented Feb 4, 2021

How far is this from being ready to go?

@arj03
Copy link
Member Author

arj03 commented Feb 4, 2021

It needs a review and then I'll probably squash the commits before merging

@arj03 arj03 requested a review from staltz February 4, 2021 15:16
index.js Show resolved Hide resolved
index.js Show resolved Hide resolved
pull.collect((err, data) => {
if (err) return cb(err)

for (let i = 0; i < data.length; ++i) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading this backwards might be a speed up, because level scans the rows in lexicographic order of the key, and we have 99% of the keys are numbers, so I think 'feed' will be the very last one. Not sure if it helps, but maybe worth putting a console.log inside the for-loop and then see whether doing it forwards or backwards is the best.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried. Didn't really change anything. The part from collect to cb is 14ms on my machine.

@KyleMaas
Copy link

KyleMaas commented Feb 5, 2021

This seems to work quite well in ssb-browser-demo. I've been clicking through it like a madman trying to get it to fail, and it just seems to keep working. And it's way faster than it was with ssb-social-index and ssb-suggest.

@arj03
Copy link
Member Author

arj03 commented Feb 5, 2021

Thanks for the review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants