startFrom parameter

[ahdinosaur]

hi, first of all, this module looks very impressive, thanks y'all for the good work. 😺

i haven't begun to use this module yet, but was scoping the usage, and had a question:

would ssb-threads methods be able to accept a startFrom parameter (a sequence number to start from), to be passed down to jitdb.operators.paginate?

when requesting Scuttlebutt threads over a non-streaming interface, i see sequence numbers being very useful as cursors. for example, this is how we did ssb-patchql. so this is what i'm looking to do for a new GraphQL interface i'm playing with. would it be possible to use sequence numbers to start the streams in ssb-threads?

cheers 💜

[staltz]

hi, first of all, this module looks very impressive, thanks y'all for the good work. smiley_cat

Thanks @ahdinosaur !

would ssb-threads methods be able to accept a startFrom parameter (a sequence number to start from), to be passed down to jitdb.operators.paginate?

Yes and no. Basically we could just pass that parameter down to jitdb.operators.paginate, but it wouldn't give you the outcome you want, because I assume what you want is to say "start from the 3rd thread", but what ssb-threads would do in jitdb is "start from the 3rd post", because the way it works it looks up posts (either replies or roots) and then weaves together a thread from that, and in the process it also removes threads from blocked peers, etc.

I don't know how GraphQL interfaces with things like this, but I believe you could also take the output pull-stream and just "pull one at a time", e.g. how I do with pull-flat-list (similar in spirit to pull-more). Can GraphQL keep the ssb-threads pull-stream output in memory? If it can, then what I just suggested would work well. If not, there is a simple method that you can apply as a post-processing step after ssb-threads returns:

pull(
  ssb.threads.publicSummary(),
  pull.skip(cursor),
  pull.take(1)
)

Where cursor is a number indicating how many you want to skip. However, this is a bit wasteful for performance (although there is a good amount of caching in ssb-db2) because the same chunk of threads will be retrieved many times from ssb-db2.

[ahdinosaur]

Basically we could just pass that parameter down to jitdb.operators.paginate, but it wouldn't give you the outcome you want, because I assume what you want is to say "start from the 3rd thread", but what ssb-threads would do in jitdb is "start from the 3rd post"

no that's good, this is what we want! 🙆

the next step is to know the latest sequence number (furthest away from the starting sequence number) processed by the query, so we could know what to ask for next (after we take some amount of items from the query and end the pull stream).

[ahdinosaur]

the original use case i have in mind first is to just control where the stream starts, so we don't fetch anything more until the user clicks through an "update" button of sorts (or pulls down for more).

but i can imagine proper pagination using sequence numbers to be a more complex beast, since posts in a thread can be all over the show. so i might need to give that more thought, and see how Piet did that in ssb-patchql.

[ahdinosaur]

yeah i'll give this more thought before i ask anything more. thanks for your response @staltz ! 🌻