PDS: Adding new leaf to MST is O(n²)

[rudyfraser]

See graph for sample size of 254 leafs being added.

Discovered this issue when testing adding records in bulk (f6b1b26). Should be able to add 1000 records in ms. Canonical code for the same test runs in 0.717 s, estimated 2 s

[DavidBuchanan314]

So just to clarify, adding n nodes is costing O(n²) overall, meaning each individual add is O(n)?

Versus the ideal expectation of O(n log n) overall, and O(log n) for each add?

[DavidBuchanan314]

I think this is your issue right here:

rsky/rsky-pds/src/repo/mst/mod.rs

Line 584 in f6b1b26

let mut new_root = MST::create(self.storage.clone(), Some(updated), Some(key_zeros))?;

I'm not a rustacean so forgive me if I'm misunderstanding, but if you're copying the whole MST storage then that's an O(n) operation (where n is the number of nodes currently in the tree)

Edit: ah yeah I'm probably mistaken, I see storage is a SqlRepoReader so that line doesn't actually copy the data

[rudyfraser]

So just to clarify, adding n nodes is costing O(n²) overall, meaning each individual add is O(n)?

Versus the ideal expectation of O(n log n) overall, and O(log n) for each add?

Yes, I think that is exactly right. I appreciate you taking a look at this!

[mackuba]

cc @steveklabnik

[steveklabnik]

Sorry for the late response here; I've glanced at this a few times, and nothing really sticks out to me. I don't have time this exact minute to try and dig in even further, but rather than look it over and go "nope, not immediately seeing it" and not leaving a comment like the last two or three times, figured I'd say something at least :)

[DavidBuchanan314]

I just came across this https://github.com/domodwyer/merkle-search-tree - which is a rust MST implementation with very impressive perf numbers. It doesn't look like it was written with atproto in mind though, so I'm not sure it'd be easy to drop in, but it might be useful for reference