New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support db2 #49
Support db2 #49
Conversation
Dictionary compression test seems to be working a bit better. Test from my browser profile seeing roughly 1000 feeds at hops 2.
The last 2 numbers are only for comparison. What they mean is that the dictionary compression got the size down from roughly 4mb json to 1.1mb. Still level is wierd:
This is not ready for review yet, just dotting down numbers as I get closer. |
Ho ho ho, turns out level is insanely naive when it comes to batches. Massive speedup incoming :) |
This last commit fixes the batching which in turn have several nice benefits:
|
After running this in the browser for a bit of testing I discovered a bug. I think the bug is actually in ssb-db2. |
Found the problem. It's a bit wierd that you can shadow a const by doing a let in a different scope :( |
How far is this from being ready to go? |
It needs a review and then I'll probably squash the commits before merging |
pull.collect((err, data) => { | ||
if (err) return cb(err) | ||
|
||
for (let i = 0; i < data.length; ++i) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reading this backwards might be a speed up, because level scans the rows in lexicographic order of the key, and we have 99% of the keys are numbers, so I think 'feed'
will be the very last one. Not sure if it helps, but maybe worth putting a console.log inside the for-loop and then see whether doing it forwards or backwards is the best.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just tried. Didn't really change anything. The part from collect to cb is 14ms on my machine.
This seems to work quite well in ssb-browser-demo. I've been clicking through it like a madman trying to get it to fail, and it just seems to keep working. And it's way faster than it was with ssb-social-index and ssb-suggest. |
Thanks for the review |
This PR adds support for db2 by checking if db2 has been loaded in secret stack and if so uses a level index in the same way it used a flumeview-reduce index before.
In this PR I also took the liberty of doing a bit of cleaning up. I converted the CI to github actions so we have tests running again here. I moved some of the glue code into a separate folder and made it possible disable them. I don't like modules doing this kind of spooky interactions, so I would rather have ssb-friends do 1 thing and that is to give a friends graph and then leave all the glueing of the modules together in another place.
I'll squash the commits for a nicer history, but first I want to put this into ssb-browser and see how it behaves.