CHATHISTORY: consider an API to discover DM correspondents

[slingamn]

This is related to, but distinct from #438. It comes out of discussion on #434 (and out-of-band with @kylef).

The previous de facto standard for retrieving history was znc.in/playback. draft/chathistory is at full feature parity with this, except for one feature: the LIST command, which "lists available buffers" (including joined channels and "active" DM correspondents).

This functionality cannot be imported directly into draft/chathistory because of the design assumption that all client synchronization state must be stored on the client side: therefore, the server has no concept of "active buffers" or "open queries". The set of DM correspondents is only defined relative to a time window.

However, the lack of such an API creates a problem: DMs from one user get "buried" behind another, so the client has to retrieve all missed DMs before it can surface new DMs of interest from previously unknown correspondents.

We can design a new API for this (and indexes to support it), it just has to be explicitly time-relative (or potentially msgid-relative?), and we have to verify that it can be efficiently implemented on the server side.

[hhirtz]

However, the lack of such an API creates a problem: DMs from one user get "buried" behind another, so the client has to retrieve all missed DMs before it can surface new DMs of interest from previously unknown correspondents.

Also, the fact that "*" is an invalid chathistory batch name i guess? or was it resolved?

[slingamn]

That's being considered on #437. I thought this merited a separate discussion thread.

[slingamn]

First pass: CHATHISTORY CORRESPONDENTS timestamp=<start> timestamp=<end> <limit>:

1. Timestamps queried and referenced in selectors are the times of the latest message sent from or to the correspondent
2. Same selector semantics as BETWEEN: starting from and excluding the first selector, ending on and excluding the second selector, either forwards or backwards in time
3. Same pagination semantics as BETWEEN
4. Returns messages like CHATHISTORY CORRESPONDENT <nick> <timestamp>. This introduces a new CHATHISTORY command; continuation is handled by BATCH as usual (this may require a new batch type?)

[slingamn]

If we can agree on an API here, discussion is trending in the direction of it replacing the * target (not totally sure about that though).

[emersion]

Looks like a good start. Some thoughts:

1. Maybe rename CORRESPONDENTS to something like LISTCORRESPONDENTS. CORRESPONDENTS and CORRESPONDENT can easily be mixed up.
2. Querying the last n correspondants is a little awkward: <start> should be set to a long time ago and <end> to a time in the future (to account for time synchronization issues between the client and the server). It would be nice to be able to set <start> and <end> to * just like LATEST, but then the server can't guess whether the other bound is a lower or upper bound.
3. It's unclear to me how pagination should work. If I want all correspondants between t1 and t2, I should first issue a command like CHATHISTORY CORRESPONDENTS timestamp=t1 timestamp=t2 100. If there are more than 100 results, what should the next command look like?

[slingamn]

Thanks!

1. Seems reasonable. Maybe I'll try and find a shorter word than "correspondent" for this...
2. I'm open to adding the same subcommands as regular CHATHISTORY here (except for AROUND, which is not relevant). As in, the previous proposal was to allow only implicit BETWEEN, but we could allow explicit BEFORE, AFTER, LATEST, and BETWEEN here. (Incidentally, the technique you described doesn't properly implement LATEST: <start> should be the present and <end> should be the distant past, in order to ensure that pagination starts at the end and counts backwards.)
3. Pagination works the same way as with regular CHATHISTORY, so your second command would be CHATHISTORY CORRESPONDENTS timestamp=t3 timestamp=t2 100, where t3 is the lowest timestamp in the first response. This has similar problems as paging through history using timestamps (if two messages or correspondents have the same millisecond-resolution timestamp, you have to choose between adjusting the interval by a millisecond and deduplicating, or potentially missing an item). So we could consider allowing msgids as well (they would be returned on the CHATHISTORY CORRESPONDENT line and accepted as selectors to CHATHISTORY LISTCORRESPONDENTS), but bouncer implementations of the spec are going to have to fall back to timestamps in some cases.

[emersion]

I'm open to adding the same subcommands as regular CHATHISTORY here

Hrm, I'd rather keep it simple if possible…

Incidentally, the technique you described doesn't properly implement LATEST: should be the present and should be the distant past, in order to ensure that pagination starts at the end and counts backwards.)

Ah, right, this stuff is confusing.

Pagination works the same way as with regular CHATHISTORY, so your second command would be CHATHISTORY CORRESPONDENTS timestamp=t3 timestamp=t2 100, where t3 is the lowest timestamp in the first response

OK, makes sense.

So we could consider allowing msgids as well (they would be returned on the CHATHISTORY CORRESPONDENT line and accepted as selectors to CHATHISTORY LISTCORRESPONDENTS), but bouncer implementations of the spec are going to have to fall back to timestamps in some cases.

Yeah. Either way, it's good to leave the door open to other criteria by prefixing the timestamp with timestamp=.

[slingamn]

Discussion continues at #450