Index performance

[arj03]

It seems to vary quite a lot between runs:

TAP version 13
# index performance
alice is @1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519
bob is @7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519
alice index feed ssb:feed/indexed-v1/BHqcHTUk856ME9NTz5LV2Mw8e2240Ly0VGyHBeMKbpM=
ok 1 wait for replication to complete
after 3000 ms bob has 2642
^C
arj@arj:~/dev/ssb-ebt$ node benchmark/index-performance.js
TAP version 13
# index performance
alice is @1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519
bob is @7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519
alice index feed ssb:feed/indexed-v1/GnMvswaMzjAOt7nnZy46b_A1rN_1Q-Xdm5wh1fSdNw4=
ok 1 wait for replication to complete
after 3000 ms bob has 532
^C
arj@arj:~/dev/ssb-ebt$ node benchmark/index-performance.js
TAP version 13
# index performance
alice is @1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519
bob is @7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519
alice index feed ssb:feed/indexed-v1/0ZqPqPGCD2E42X6ZyWqqdHMCXNlV-TqFCopaxC5VjPU=
ok 1 wait for replication to complete
after 3000 ms bob has 3140

[arj03]

And those messages are index + main feed.

For comparison with just main feed replication:

TAP version 13
# butt2 performance
# classic performance
adding 15k msgs: 3.779s
ok 1 wait for replication to complete
after 3000 ms bob has 6670

[arj03]

I was actually expecting this to be a lot slower. So mayyyybeeee it's good enough if we only replicate small about index feeds.

[staltz]

I was actually expecting this to be a lot slower. So mayyyybeeee it's good enough if we only replicate small about index feeds.

There's two cases:

1. Person who already replicated Alice's main feed will now begin replicating her about index feed, causing ssb.db.get to return valid messages
2. Person who never replicated Alice's main feed will now begin replicating her about index feed, causing ssb.db.get to return empty and thus addTransaction

In any case, even if it's a small amount of messages (let's say 5 about msgs per person), because we'll be replicating a lot of these about index feeds, say for all the hops 2 and hops 3 accounts, it might be 20k persons, so 100k messages. Times 2 because payload+index msg so 200k messages. Maybe this is crazy because not everyone has that many abouts, but still it could be that total amount of abouts is not that small.

[arj03]

Yeah, in any case now we have baseline test and can improve things.

I checked how many about messages different feeds have and there is a big difference between newer and old ones. Old often have 100-200 with around 10-15 self assigned about messages (worst is mix with 750 :-)), while newer have around 2-5.

[staltz]

Let's get it merged?

[arj03]

Thanks for the review 🙂