What is the complexity/performance characteristics of query, copy, and rename?

[hossameldeen]

Hi there :)

I'm trying to get my head around what to expect performance-wise from beaker.hyperdrive.* methods.

(1) What is the order of the operations in query? ... In specific, is it like SQL in that sort is run before limit?
Relevant: https://wizardzines.com/zines/sql/samples/from.png

(2) If so, how does Beaker/hyperdrive achieve that in an efficient way? ... Does it, by default, maintain index on, e.g., ctime, or does it go through all the files in a directory?

(3) In general, should I deal with query as if there're any default indexes or should I deal with it as if it'll just scan all the files in a directory (or directories) and then filter?

(4) Regarding copy and rename, are they O(1) or O(total number of descendants in the directory)?

Thanks in advance and thank you so much for your time :)
Sorry if the answer is already in the docs. To be honest, I just skimmed them xD

[pfrazee]

@hossameldeen No need to apologize! These are good questions to ask.

1. You can see how query is implemented here. It basically works by doing a series of folder-scans to satisfy the path-globs, merges, sorts, and then slices.
2. There is not currently any secondary-indexing. @mafintosh mentioned there's an API update to hypertrie coming which might make order-by-mtime naturally fast (basically using the log order as the index) but we're going to need more than that. We're still discussing how we want to approach the secondary indexes; there are a lot of interesting options to explore.
3. Currently they're all scan & filters. When building an app, I think the best strategy for scaling is to use local storage or indexeddb to store cached results and then run query() in the background to update those cached results.
4. Currently they're O(n) but I believe maf and andrew have an update which will make them O(1) and atomic.

[hossameldeen]

Sorry for late reply and thank you for the great answer! (And for the Monaco module issue :-) )

Ummm ... regarding (2): So, it's not possible/efficient at the moment to provide history() method like the one in DatArchive?

[pfrazee]

2 is really more a statement about getting an efficient current state order by mtime.

I'm not sure if it's possible to get history() back in, but you can use diff() to get similar information (though it's a little harder)