From f4aa9230a057bb41d18a74468b419a3230b321f9 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 24 Feb 2021 19:57:11 -0600 Subject: [PATCH 01/28] Initial MSC draft for jump to date --- proposals/3030-jump-to-date.md | 73 ++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 proposals/3030-jump-to-date.md diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md new file mode 100644 index 00000000000..199207afc02 --- /dev/null +++ b/proposals/3030-jump-to-date.md @@ -0,0 +1,73 @@ +# MSC3030: Jump to date API endpoint + +Add an API that makes it easy to find the closest messages for a given timestamp. + +The goal of this change is to have clients be able to implement a jump to date feature in order to see messages back at a given point in time. Pick a date from a calender, heatmap, or paginate next/previous between days and view all of the messages that were sent on that date. + +For our [roadmap of feature parity with Gitter](https://github.com/vector-im/roadmap/issues/26), we're also interested in using this for a new better static Matrix archive. Our idea is to server-side render [Hydrogen](https://github.com/vector-im/hydrogen-web) and this new endpoint would allow us to jump back on the fly without having to paginate and keep track of everything in order to display the selected date. + +Also useful for archiving and backup use cases. This new endpoint can be used to slice the messages by day and persist to file. + +Related issue: [*URL for an arbitrary day of history and navigation for next and previous days* (vector-im/element-web#7677)](https://github.com/vector-im/element-web/issues/7677) + + +## Problem + +These types of use cases are not supported by the current Matrix API because it has no way to fetch or filter older messages besides a manual brute force pagination from the latest. Paginating is time-consuming and expensive to process every event as you go (not practical for clients). Imagine wanting to get a message from 3 years ago 😫 + + +## Proposal + +Add the `?around=` query parameter to the `GET /_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the response at the message with `origin_server_ts` closest to the provided `around` timestamp. The direction is determined by the existing `?dir` query parameter. + +Use topoligical ordering, just as Element would use if you follow a permalink. + + +## Potential issues + +If you ask for β€œthe message with `origin_server_ts` closest to Jan 1st 2018” you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. + + +## Alternatives + +### Filter by date in `RoomEventFilter` + +Extend `RoomEventFilter` to be able to specify a timestamp or a date range. The `RoomEventFilter` can be passed via the `?filter` query param on the `/messages` endpoint. + + +### New `destination_server_ts` field + +Add a new field and index on messages called `destination_server_ts` which indicates when the message was received from federation. This gives a more "real" time for how someone would actually consume those messages. + +The contract of the API is "show me messages my server received at time T" rather than the messy confusion of showing a delayed message which happened to originally be sent at time T. + +We've decided against this approach because the backfill from federated servers could be horribly late. + +--- + +Related issue around `/sync` vs `/messages`, https://github.com/matrix-org/synapse/issues/7164 + +> Sync returns things in the order they arrive at the server; backfill returns them in the order determined by the event graph. +> +> *-- @richvdh, https://github.com/matrix-org/synapse/issues/7164#issuecomment-605877176* + +> The general idea is that, if you're following a room in real-time (ie, `/sync`), you probably want to see the messages as they arrive at your server, rather than skipping any that arrived late; whereas if you're looking at a historical section of timeline (ie, `/messages`), you want to see the best representation of the state of the room as others were seeing it at the time. +> +> *-- @richvdh , https://github.com/matrix-org/synapse/issues/7164#issuecomment-605953296* + + +## Security considerations + +We're only going to expose messages according to the existing message history setting in the room (`m.room.history_visibility`). No extra data is exposed, just a new way to sort through it all. + + + +## Unstable prefix + +*If a proposal is implemented before it is included in the spec, then implementers must ensure that the +implementation is compatible with the final version that lands in the spec. This generally means that +experimental implementations should use `/unstable` endpoints, and use vendor prefixes where necessary. +For more information, see [MSC2324](https://github.com/matrix-org/matrix-doc/pull/2324). This section +should be used to document things such as what endpoints and names are being used while the feature is +in development, the name of the unstable feature flag to use to detect support for the feature, or what +migration steps are needed to switch to newer versions of the proposal.* From 11025d6f30959ec3d3a2fa771f8f3d582b3a5de6 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 17 Nov 2021 01:46:09 -0600 Subject: [PATCH 02/28] Update with alternate /timestamp_to_event endpoint --- proposals/3030-jump-to-date.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 199207afc02..dc9a52b2d08 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -18,18 +18,32 @@ These types of use cases are not supported by the current Matrix API because it ## Proposal -Add the `?around=` query parameter to the `GET /_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the response at the message with `origin_server_ts` closest to the provided `around` timestamp. The direction is determined by the existing `?dir` query parameter. -Use topoligical ordering, just as Element would use if you follow a permalink. +Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest event to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. + +In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET /_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other homeservers can use to ask about their closest event to the timestamp. + +The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver implementation. Although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. + +--- + +In order to paginate `/messages`, we needa pagination token which we can get using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limi=0` for the `event_id` returned by `/timestamp_to_event`. ## Potential issues -If you ask for β€œthe message with `origin_server_ts` closest to Jan 1st 2018” you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. +If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. ## Alternatives + +### Paginate `/messages` from timestamp + +Add the `?around=` query parameter to the `GET /_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the response at the message with `origin_server_ts` closest to the provided `around` timestamp. The direction is determined by the existing `?dir` query parameter. + +Use topoligical ordering, just as Element would use if you follow a permalink. + ### Filter by date in `RoomEventFilter` Extend `RoomEventFilter` to be able to specify a timestamp or a date range. The `RoomEventFilter` can be passed via the `?filter` query param on the `/messages` endpoint. From 95cd693b80b7f2888e0e2040827c4f49c8677615 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 30 Nov 2021 01:54:19 -0600 Subject: [PATCH 03/28] Add origin_server_ts for quick remote to local comparison As discussed at https://github.com/matrix-org/synapse/pull/9445#discussion_r757098009 --- proposals/3030-jump-to-date.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index dc9a52b2d08..dec1acb2948 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -19,15 +19,31 @@ These types of use cases are not supported by the current Matrix API because it ## Proposal -Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest event to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. +Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. -In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET /_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other homeservers can use to ask about their closest event to the timestamp. +In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET /_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the local one. The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver implementation. Although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. +``` +GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_event?ts=&dir= +{ + "event_id": ... +} +``` + +Federation API endpoint: +``` +GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/?ts=&dir= +{ + "event_id": ... + "origin_server_ts": ... +} +``` + --- -In order to paginate `/messages`, we needa pagination token which we can get using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limi=0` for the `event_id` returned by `/timestamp_to_event`. +In order to paginate `/messages`, we need a pagination token which we can get using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limi=0` for the `event_id` returned by `/timestamp_to_event`. ## Potential issues From 13910e7643544eef8bdfd5a25f5c84eb24998bc3 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 30 Nov 2021 02:04:42 -0600 Subject: [PATCH 04/28] Add origin_server_ts to client endpoint --- proposals/3030-jump-to-date.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index dec1acb2948..a242b8656aa 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -19,7 +19,7 @@ These types of use cases are not supported by the current Matrix API because it ## Proposal -Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. +Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the `event_id` fetched is too far out of range to be useful for your use case. In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET /_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the local one. @@ -29,6 +29,7 @@ The heuristics for deciding when to ask another homeserver for a closer event if GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_event?ts=&dir= { "event_id": ... + "origin_server_ts": ... } ``` From 2559770eb720e0fab6ce02fb48e67100428f97bb Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 3 Dec 2021 14:17:03 -0600 Subject: [PATCH 05/28] Wrap lines --- proposals/3030-jump-to-date.md | 99 ++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 22 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index a242b8656aa..b82ad2b1a06 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -1,29 +1,61 @@ # MSC3030: Jump to date API endpoint -Add an API that makes it easy to find the closest messages for a given timestamp. +Add an API that makes it easy to find the closest messages for a given +timestamp. -The goal of this change is to have clients be able to implement a jump to date feature in order to see messages back at a given point in time. Pick a date from a calender, heatmap, or paginate next/previous between days and view all of the messages that were sent on that date. +The goal of this change is to have clients be able to implement a jump to date +feature in order to see messages back at a given point in time. Pick a date from +a calender, heatmap, or paginate next/previous between days and view all of the +messages that were sent on that date. -For our [roadmap of feature parity with Gitter](https://github.com/vector-im/roadmap/issues/26), we're also interested in using this for a new better static Matrix archive. Our idea is to server-side render [Hydrogen](https://github.com/vector-im/hydrogen-web) and this new endpoint would allow us to jump back on the fly without having to paginate and keep track of everything in order to display the selected date. +For our [roadmap of feature parity with +Gitter](https://github.com/vector-im/roadmap/issues/26), we're also interested +in using this for a new better static Matrix archive. Our idea is to server-side +render [Hydrogen](https://github.com/vector-im/hydrogen-web) and this new +endpoint would allow us to jump back on the fly without having to paginate and +keep track of everything in order to display the selected date. -Also useful for archiving and backup use cases. This new endpoint can be used to slice the messages by day and persist to file. +Also useful for archiving and backup use cases. This new endpoint can be used to +slice the messages by day and persist to file. -Related issue: [*URL for an arbitrary day of history and navigation for next and previous days* (vector-im/element-web#7677)](https://github.com/vector-im/element-web/issues/7677) +Related issue: [*URL for an arbitrary day of history and navigation for next and +previous days* +(vector-im/element-web#7677)](https://github.com/vector-im/element-web/issues/7677) ## Problem -These types of use cases are not supported by the current Matrix API because it has no way to fetch or filter older messages besides a manual brute force pagination from the latest. Paginating is time-consuming and expensive to process every event as you go (not practical for clients). Imagine wanting to get a message from 3 years ago 😫 +These types of use cases are not supported by the current Matrix API because it +has no way to fetch or filter older messages besides a manual brute force +pagination from the latest. Paginating is time-consuming and expensive to +process every event as you go (not practical for clients). Imagine wanting to +get a message from 3 years ago 😫 ## Proposal -Add new client API endpoint `GET /_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the `event_id` fetched is too far out of range to be useful for your use case. - -In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET /_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the local one. - -The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver implementation. Although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. +Add new client API endpoint `GET +/_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` +which fetches the closest `event_id` to the given timestamp `ts` query parameter +in the direction specified by the `dir` query parameter. This endpoint also +returns `origin_server_ts` to make it easy to do a quick comparison to see if +the `event_id` fetched is too far out of range to be useful for your use case. + +In order to solve the problem where a remote federated homeserver does not have +all of the history in a room and no suitably close event, we also add a server +API endpoint `GET +/_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` +which other homeservers can use to ask about their closest `event_id` to the +timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a +quick comparison to see if the remote `event_id` fetched is closer than the +local one. + +The heuristics for deciding when to ask another homeserver for a closer event if +your homeserver doesn't have something close, is left up to the homeserver +implementation. Although the heuristics will probably be based on whether the +closest event is a forward/backward extremity indicating it's next to a gap of +events which are potentially closer. ``` GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_event?ts=&dir= @@ -44,12 +76,17 @@ GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/? --- -In order to paginate `/messages`, we need a pagination token which we can get using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limi=0` for the `event_id` returned by `/timestamp_to_event`. +In order to paginate `/messages`, we need a pagination token which we can get +using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limi=0` for the +`event_id` returned by `/timestamp_to_event`. ## Potential issues -If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. +If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you +might actually get a rogue random delayed one that was backfilled from a +federated server, but the human can figure that out by trying again with a +slight variation on the date or something. ## Alternatives @@ -57,39 +94,57 @@ If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you ### Paginate `/messages` from timestamp -Add the `?around=` query parameter to the `GET /_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the response at the message with `origin_server_ts` closest to the provided `around` timestamp. The direction is determined by the existing `?dir` query parameter. +Add the `?around=` query parameter to the `GET +/_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the +response at the message with `origin_server_ts` closest to the provided `around` +timestamp. The direction is determined by the existing `?dir` query parameter. Use topoligical ordering, just as Element would use if you follow a permalink. ### Filter by date in `RoomEventFilter` -Extend `RoomEventFilter` to be able to specify a timestamp or a date range. The `RoomEventFilter` can be passed via the `?filter` query param on the `/messages` endpoint. +Extend `RoomEventFilter` to be able to specify a timestamp or a date range. The +`RoomEventFilter` can be passed via the `?filter` query param on the `/messages` +endpoint. ### New `destination_server_ts` field -Add a new field and index on messages called `destination_server_ts` which indicates when the message was received from federation. This gives a more "real" time for how someone would actually consume those messages. +Add a new field and index on messages called `destination_server_ts` which +indicates when the message was received from federation. This gives a more +"real" time for how someone would actually consume those messages. -The contract of the API is "show me messages my server received at time T" rather than the messy confusion of showing a delayed message which happened to originally be sent at time T. +The contract of the API is "show me messages my server received at time T" +rather than the messy confusion of showing a delayed message which happened to +originally be sent at time T. -We've decided against this approach because the backfill from federated servers could be horribly late. +We've decided against this approach because the backfill from federated servers +could be horribly late. --- -Related issue around `/sync` vs `/messages`, https://github.com/matrix-org/synapse/issues/7164 +Related issue around `/sync` vs `/messages`, +https://github.com/matrix-org/synapse/issues/7164 -> Sync returns things in the order they arrive at the server; backfill returns them in the order determined by the event graph. +> Sync returns things in the order they arrive at the server; backfill returns +> them in the order determined by the event graph. > > *-- @richvdh, https://github.com/matrix-org/synapse/issues/7164#issuecomment-605877176* -> The general idea is that, if you're following a room in real-time (ie, `/sync`), you probably want to see the messages as they arrive at your server, rather than skipping any that arrived late; whereas if you're looking at a historical section of timeline (ie, `/messages`), you want to see the best representation of the state of the room as others were seeing it at the time. +> The general idea is that, if you're following a room in real-time (ie, +> `/sync`), you probably want to see the messages as they arrive at your server, +> rather than skipping any that arrived late; whereas if you're looking at a +> historical section of timeline (ie, `/messages`), you want to see the best +> representation of the state of the room as others were seeing it at the time. > > *-- @richvdh , https://github.com/matrix-org/synapse/issues/7164#issuecomment-605953296* ## Security considerations -We're only going to expose messages according to the existing message history setting in the room (`m.room.history_visibility`). No extra data is exposed, just a new way to sort through it all. +We're only going to expose messages according to the existing message history +setting in the room (`m.room.history_visibility`). No extra data is exposed, +just a new way to sort through it all. From f19a43ab51d202633ce121ef576d493adb420461 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 3 Dec 2021 14:35:39 -0600 Subject: [PATCH 06/28] Use stable when discussing MSC and document unstable --- proposals/3030-jump-to-date.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index b82ad2b1a06..88c291e9ccc 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -36,7 +36,7 @@ get a message from 3 years ago 😫 Add new client API endpoint `GET -/_matrix/client/r0/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` +/_matrix/client/v1/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if @@ -58,7 +58,7 @@ closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. ``` -GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_event?ts=&dir= +GET /_matrix/client/v1/rooms//timestamp_to_event?ts=&dir= { "event_id": ... "origin_server_ts": ... @@ -67,7 +67,7 @@ GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_even Federation API endpoint: ``` -GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/?ts=&dir= +GET /_matrix/federation/v1/timestamp_to_event/?ts=&dir= { "event_id": ... "origin_server_ts": ... @@ -150,10 +150,20 @@ just a new way to sort through it all. ## Unstable prefix -*If a proposal is implemented before it is included in the spec, then implementers must ensure that the -implementation is compatible with the final version that lands in the spec. This generally means that -experimental implementations should use `/unstable` endpoints, and use vendor prefixes where necessary. -For more information, see [MSC2324](https://github.com/matrix-org/matrix-doc/pull/2324). This section -should be used to document things such as what endpoints and names are being used while the feature is -in development, the name of the unstable feature flag to use to detect support for the feature, or what -migration steps are needed to switch to newer versions of the proposal.* +While this MSC is not considered stable, the endpoints are available at `/unstable/org.matrix.msc3030` instead of their `/v1` description from above. + +``` +GET /_matrix/client/unstable/org.matrix.msc3030/rooms//timestamp_to_event?ts=&dir= +{ + "event_id": ... + "origin_server_ts": ... +} +``` + +``` +GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/?ts=&dir= +{ + "event_id": ... + "origin_server_ts": ... +} +``` From f87a4f2ee7787721e47fbee0efe728b5be685676 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 3 Dec 2021 14:46:01 -0600 Subject: [PATCH 07/28] Describe the direction parameter --- proposals/3030-jump-to-date.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 88c291e9ccc..d9a95e14511 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -38,9 +38,11 @@ get a message from 3 years ago 😫 Add new client API endpoint `GET /_matrix/client/v1/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter -in the direction specified by the `dir` query parameter. This endpoint also -returns `origin_server_ts` to make it easy to do a quick comparison to see if -the `event_id` fetched is too far out of range to be useful for your use case. +in the direction specified by the `dir` query parameter. The direction `dir` +query parameter accepts `f` for forward-in-time from the timestamp and `b` for +backward-in-time from the timestamp. This endpoint also returns +`origin_server_ts` to make it easy to do a quick comparison to see if the +`event_id` fetched is too far out of range to be useful for your use case. In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server From d9b0bed8f976350e28d3d670e983304704c0e1aa Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 15 Dec 2021 05:23:02 -0600 Subject: [PATCH 08/28] Add server support detection --- proposals/3030-jump-to-date.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index d9a95e14511..a4539df7c99 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -169,3 +169,7 @@ GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/? "origin_server_ts": ... } ``` + +Servers will indicate support for the new endpoint via a non-empty value for feature flag +`org.matrix.msc3030` in `unstable_features` in the response to `GET +/_matrix/client/versions`. From cc6a4a372938c52df41b740919fe1a01679c9f53 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 18 Jan 2022 01:19:51 -0600 Subject: [PATCH 09/28] Fix typos --- proposals/3030-jump-to-date.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index a4539df7c99..7e4aa68e2ab 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -79,7 +79,7 @@ GET /_matrix/federation/v1/timestamp_to_event/?ts=&dir=` query parameter to the `GET response at the message with `origin_server_ts` closest to the provided `around` timestamp. The direction is determined by the existing `?dir` query parameter. -Use topoligical ordering, just as Element would use if you follow a permalink. +Use topological ordering, just as Element would use if you follow a permalink. ### Filter by date in `RoomEventFilter` From 2c8cdd845a2d4bb80c413e009a7657c923f8b66e Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 18 Jan 2022 20:22:43 -0600 Subject: [PATCH 10/28] Explain what happens when an event can't be found Fix https://github.com/matrix-org/matrix-doc/pull/3030#discussion_r787002549 --- proposals/3030-jump-to-date.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 7e4aa68e2ab..7b648ae03f6 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -44,6 +44,10 @@ backward-in-time from the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the `event_id` fetched is too far out of range to be useful for your use case. +When an event can't be found, the endpoint throws a 404 +`"errcode":"M_NOT_FOUND",` (example error message `"error":"Unable to find +event from 1672531200000 in direction f"`). + In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server API endpoint `GET From 9aa73f4dd751bbfb071a72353fac58b76a5836f6 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 18 Jan 2022 20:30:21 -0600 Subject: [PATCH 11/28] Add context behind why we chose /timestamp_to_event vs alternatives Fix https://github.com/matrix-org/matrix-doc/pull/3030#discussion_r785425438 --- proposals/3030-jump-to-date.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 7b648ae03f6..0c51314a6d7 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -44,9 +44,9 @@ backward-in-time from the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the `event_id` fetched is too far out of range to be useful for your use case. -When an event can't be found, the endpoint throws a 404 -`"errcode":"M_NOT_FOUND",` (example error message `"error":"Unable to find -event from 1672531200000 in direction f"`). +When an event can't be found in the given direction, the endpoint throws a 404 +`"errcode":"M_NOT_FOUND",` (example error message `"error":"Unable to find event +from 1672531200000 in direction f"`). In order to solve the problem where a remote federated homeserver does not have all of the history in a room and no suitably close event, we also add a server @@ -97,6 +97,14 @@ slight variation on the date or something. ## Alternatives +We chose the current `/timestamp_to_event` route because it sounded like the +easist path forward to bring it to fruition and get some real-world experience. +And was on our mind during the [initial discussion](https://docs.google.com/document/d/1KCEmpnGr4J-I8EeaVQ8QJZKBDu53ViI7V62y5BzfXr0/edit#bookmark=id.qu9k9wje9pxm) because there was some prior art with a [WIP +implementation](https://github.com/matrix-org/synapse/pull/9445/commits/91b1b3606c9fb9eede0a6963bc42dfb70635449f) +from @erikjohnston. The alternatives haven't been thrown out for a particular +reason and we could still go down those routes depending on how people like the +current design. + ### Paginate `/messages` from timestamp From fdd0022de8ef3099fc5eeb64ce2bf1b33b7ab4e4 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Mon, 24 Jan 2022 17:36:07 -0600 Subject: [PATCH 12/28] Add comments about authentication and rate-limiting Fix https://github.com/matrix-org/matrix-doc/pull/3030#discussion_r786351083 --- proposals/3030-jump-to-date.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 0c51314a6d7..ef44d61b85e 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -63,6 +63,9 @@ implementation. Although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. +These endpoints are authenticated and should be rate-limited like similar client +and federation endpoints to prevent resource exhaustion abuse. + ``` GET /_matrix/client/v1/rooms//timestamp_to_event?ts=&dir= { From 8238dfeeae58de3ce1ead0191684fe4589a073a3 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 8 Apr 2022 22:28:23 -0500 Subject: [PATCH 13/28] Return pagination token directly in future iteration See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r787297190 --- proposals/3030-jump-to-date.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index ef44d61b85e..a98ee7ece9f 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -34,7 +34,6 @@ get a message from 3 years ago 😫 ## Proposal - Add new client API endpoint `GET /_matrix/client/v1/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter @@ -89,6 +88,9 @@ In order to paginate `/messages`, we need a pagination token which we can get using `GET /_matrix/client/r0/rooms/{roomId}/context/{eventId}?limit=0` for the `event_id` returned by `/timestamp_to_event`. +We can always iterate on `/timestamp_to_event` later and return a pagination +token directly in another MSC ⏩ + ## Potential issues From 75c157b58bcf8823972cd3fa106ae37e9ccc9511 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 8 Apr 2022 22:32:34 -0500 Subject: [PATCH 14/28] Abuse /timestamp_to_event to get create event As suggested by @turt2live, https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r846444317 --- proposals/3030-jump-to-date.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index a98ee7ece9f..a78aa8ac20b 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -99,6 +99,12 @@ might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. +Clients could abuse this new API for getting the `m.room.create` event, so +servers might want to put extra care into optimizing whatever lookups they do. +The create event contains quite a lot of information that a client needs in +order to operate, so it is frequently requested by said clients. For example, +the room type and room version (for displaying warnings about stability). + ## Alternatives From cbd388fc414ee13d498f52aadaa12f009a7a0153 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Fri, 8 Apr 2022 22:49:12 -0500 Subject: [PATCH 15/28] Unrenderable events As proposed by @turt2live, https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r846447351 --- proposals/3030-jump-to-date.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index a78aa8ac20b..91b0ba46428 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -94,11 +94,40 @@ token directly in another MSC ⏩ ## Potential issues + +### Receiving a rogue random delayed event ID + If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. + +### Receiving an unrenderable event ID + +Another issue is that clients could land on an event they can't/won't render, +such as a reaction, then they'll be forced to desperately seek around the +timeline until they find an event they can do something with. + +Eg: + - Client wants to jump to January 1st, 2022 + - Server says there's an event on January 2nd, 2022 that is close enough + - Client finds out there's a ton of unrenderable events like memberships, poll responses, reactions, etc at that time + - Client starts paginating forwards, finally finding an event on January 27th it can render + - Client wasn't aware that the actual nearest neighbouring event was backwards on December 28th, 2021 because it didn't paginate in that direction + - User is confused that they are a month past the target date when the message is *right there*. + +Clients can be smarter here though. Clients can see when events were sent as +they paginate and if they see they're going more than a couple days out, they +can also try the other direction before going further and further away. + +Clients can also just explain to the user what happened with a little toast: "We +were unable to find an event to display on January 1st, 2022. The closest event +after that date is on January 27th." + + +### Abusing the `/timestamp_to_event` API to get the `m.room.create` event + Clients could abuse this new API for getting the `m.room.create` event, so servers might want to put extra care into optimizing whatever lookups they do. The create event contains quite a lot of information that a client needs in From bb732d9ddaaaadf654152e5ed6ad71e5c450e02d Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Sat, 9 Apr 2022 00:16:30 -0500 Subject: [PATCH 16/28] Add some complication thoughts around alternatives Context: https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r846449709 --- proposals/3030-jump-to-date.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 91b0ba46428..2740a9708aa 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -146,7 +146,7 @@ reason and we could still go down those routes depending on how people like the current design. -### Paginate `/messages` from timestamp +### Paginate `/messages?around=` from timestamp Add the `?around=` query parameter to the `GET /_matrix/client/r0/rooms/{roomId}/messages` endpoint. This will start the @@ -155,12 +155,28 @@ timestamp. The direction is determined by the existing `?dir` query parameter. Use topological ordering, just as Element would use if you follow a permalink. +This alternative could be confusing to the end-user around how this plays with +the existing query parameters +`/messages?from={paginationToken}&to={paginationToken}` which also determine +what part of the timeline to query. Those parameters could be extended to accept +timestamps in addition to pagination tokens but then could get confusing again +when you start mixing timestamps and pagination tokens. The homeserver also has +to disambiguate what a pagination token looks like vs a unix timestamp. Since +pagination tokens don't follow a certain convention, some homeserver +implementations may already be using arbitrary number tokens already which would +be impossible to distinguish from a timestamp. + + ### Filter by date in `RoomEventFilter` Extend `RoomEventFilter` to be able to specify a timestamp or a date range. The `RoomEventFilter` can be passed via the `?filter` query param on the `/messages` endpoint. +This suffers from the same confusion to the end-user of how it plays with how +this plays with `/messages?from={paginationToken}&to={paginationToken}` which +also determines what part of the timeline to query. + ### New `destination_server_ts` field From 4d2a45ad5ccc1c095bf1a2468577dbaef6287437 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Sat, 9 Apr 2022 00:28:43 -0500 Subject: [PATCH 17/28] Backfill event so we can get pagination token See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r846578171 --- proposals/3030-jump-to-date.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 2740a9708aa..b97c3852707 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -27,9 +27,9 @@ previous days* These types of use cases are not supported by the current Matrix API because it has no way to fetch or filter older messages besides a manual brute force -pagination from the latest. Paginating is time-consuming and expensive to -process every event as you go (not practical for clients). Imagine wanting to -get a message from 3 years ago 😫 +pagination from the most recent event in the room. Paginating is time-consuming +and expensive to process every event as you go (not practical for clients). +Imagine wanting to get a message from 3 years ago 😫 ## Proposal @@ -54,7 +54,10 @@ API endpoint `GET which other homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the -local one. +local one. After the local homeserver receives a response from the federation +endpoint, it should probably should try to backfill this event so that it's +available to query with `/context` from a client in order to get a pagination +token. The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver @@ -94,7 +97,6 @@ token directly in another MSC ⏩ ## Potential issues - ### Receiving a rogue random delayed event ID If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you From 38b81475a42bfbdf870c97b3ad9ef3e0dbd8cf2b Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 14 Sep 2022 03:02:55 -0500 Subject: [PATCH 18/28] Heuristic for which server to try first See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r967574944 --- proposals/3030-jump-to-date.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index b97c3852707..ee91215faf1 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -65,6 +65,10 @@ implementation. Although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. +A good heuristic for which servers to try first is to sort by servers that have +been in the room the longest because they're most likely to have anything we ask +about. + These endpoints are authenticated and should be rate-limited like similar client and federation endpoints to prevent resource exhaustion abuse. From 067bdeb6fb8b6aa7e8c5b66138bdc1c4b050ba12 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 14 Sep 2022 03:05:33 -0500 Subject: [PATCH 19/28] Give a suggestion on where to backfill from See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r967574843 --- proposals/3030-jump-to-date.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index ee91215faf1..46fae6de931 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -55,9 +55,9 @@ which other homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the local one. After the local homeserver receives a response from the federation -endpoint, it should probably should try to backfill this event so that it's -available to query with `/context` from a client in order to get a pagination -token. +endpoint, it should probably should try to backfill this event via the +federation `/event/` endpoint so that it's available to query with +`/context` from a client in order to get a pagination token. The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver From 1804b713b1ec1812a3e97a87234beae60607a4af Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 14 Sep 2022 03:20:30 -0500 Subject: [PATCH 20/28] Add alternative suggestion from @alphapapa See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r868478333 --- proposals/3030-jump-to-date.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 46fae6de931..5d2d4a9e206 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -172,6 +172,15 @@ pagination tokens don't follow a certain convention, some homeserver implementations may already be using arbitrary number tokens already which would be impossible to distinguish from a timestamp. +A related alternative is to use `/messages` with a `from_time`/`to_time` (or +`from_ts`/`to_ts`) query parameters that only accept timestamps which solves the +confusion and disambigution problem of trying to re-use the existing `from`/`to` +query paramters. Re-using `/messages` would reduce the number of round-trips and +potentially client-side implementations for the use case where you want to fetch +a window of messages from a given time. But has the same round-trip problem if +you want to use the returned `event_id` with `/context` or another endpoint +instead. + ### Filter by date in `RoomEventFilter` From 8ca7a0873b580f8f61b1b6a241349b183f068f0b Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 11 Oct 2022 18:34:47 -0500 Subject: [PATCH 21/28] Better wording and fix typo Co-authored-by: Travis Ralston --- proposals/3030-jump-to-date.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 5d2d4a9e206..4e5171fb1c3 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -8,7 +8,7 @@ feature in order to see messages back at a given point in time. Pick a date from a calender, heatmap, or paginate next/previous between days and view all of the messages that were sent on that date. -For our [roadmap of feature parity with +Alongside the [roadmap of feature parity with Gitter](https://github.com/vector-im/roadmap/issues/26), we're also interested in using this for a new better static Matrix archive. Our idea is to server-side render [Hydrogen](https://github.com/vector-im/hydrogen-web) and this new @@ -35,7 +35,7 @@ Imagine wanting to get a message from 3 years ago 😫 ## Proposal Add new client API endpoint `GET -/_matrix/client/v1/rooms/{roomId}/timestamp_to_event?ts=?dir=[f|b]` +/_matrix/client/v1/rooms/{roomId}/timestamp_to_event?ts=&dir=[f|b]` which fetches the closest `event_id` to the given timestamp `ts` query parameter in the direction specified by the `dir` query parameter. The direction `dir` query parameter accepts `f` for forward-in-time from the timestamp and `b` for From ad99b643360acda1dc8598b5d201d93bcff64442 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 11 Oct 2022 18:42:24 -0500 Subject: [PATCH 22/28] No difference in homeservers See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r992858188 --- proposals/3030-jump-to-date.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 4e5171fb1c3..bb33f56089c 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -47,17 +47,15 @@ When an event can't be found in the given direction, the endpoint throws a 404 `"errcode":"M_NOT_FOUND",` (example error message `"error":"Unable to find event from 1672531200000 in direction f"`). -In order to solve the problem where a remote federated homeserver does not have -all of the history in a room and no suitably close event, we also add a server -API endpoint `GET -/_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` -which other homeservers can use to ask about their closest `event_id` to the -timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a -quick comparison to see if the remote `event_id` fetched is closer than the -local one. After the local homeserver receives a response from the federation -endpoint, it should probably should try to backfill this event via the -federation `/event/` endpoint so that it's available to query with -`/context` from a client in order to get a pagination token. +In order to solve the problem where a homeserver does not have all of the history in a +room and no suitably close event, we also add a server API endpoint `GET +/_matrix/federation/v1/timestamp_to_event/{roomId}?ts=?dir=[f|b]` which other +homeservers can use to ask about their closest `event_id` to the timestamp. This +endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see +if the remote `event_id` fetched is closer than the local one. After the local +homeserver receives a response from the federation endpoint, it should probably should +try to backfill this event via the federation `/event/` endpoint so that it's +available to query with `/context` from a client in order to get a pagination token. The heuristics for deciding when to ask another homeserver for a closer event if your homeserver doesn't have something close, is left up to the homeserver From bc2be7828a1750899ce0c942bf3e216f4e162668 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 19 Oct 2022 12:17:02 -0500 Subject: [PATCH 23/28] Fix typos Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/3030-jump-to-date.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index bb33f56089c..5e955304475 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -58,8 +58,8 @@ try to backfill this event via the federation `/event/` endpoint so th available to query with `/context` from a client in order to get a pagination token. The heuristics for deciding when to ask another homeserver for a closer event if -your homeserver doesn't have something close, is left up to the homeserver -implementation. Although the heuristics will probably be based on whether the +your homeserver doesn't have something close, are left up to the homeserver +implementation, although the heuristics will probably be based on whether the closest event is a forward/backward extremity indicating it's next to a gap of events which are potentially closer. From 23cfab00f7be10ea57c4f7fc52dcd58051c81e00 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 19 Oct 2022 13:56:14 -0500 Subject: [PATCH 24/28] Fix extra word typo --- proposals/3030-jump-to-date.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 5e955304475..f6912890e00 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -53,7 +53,7 @@ room and no suitably close event, we also add a server API endpoint `GET homeservers can use to ask about their closest `event_id` to the timestamp. This endpoint also returns `origin_server_ts` to make it easy to do a quick comparison to see if the remote `event_id` fetched is closer than the local one. After the local -homeserver receives a response from the federation endpoint, it should probably should +homeserver receives a response from the federation endpoint, it probably should try to backfill this event via the federation `/event/` endpoint so that it's available to query with `/context` from a client in order to get a pagination token. From 509c1b4e51a03adafc6264caa21fe14da1f743c2 Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Wed, 19 Oct 2022 14:24:56 -0500 Subject: [PATCH 25/28] Summarizing discussion around why `dir` instead of closest See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r849310414 --- proposals/3030-jump-to-date.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index f6912890e00..3681b3fed73 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -191,6 +191,38 @@ this plays with `/messages?from={paginationToken}&to={paginationToken}` which also determines what part of the timeline to query. +### Return the closest event in any direction + +The question you may be asking is "What's the rationale for including a `dir`? Can't we +just return the closest/nearest event regardless of direction?" Or at least add a +`dir=c` for the option to find the closest event. + +For clients, it doesn't make that much of a difference because you can always compare +the timestamp you made the request with to the `origin_server_ts` that was returned to +see what direction you need to paginate further in. And it's a +[toss-up](https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r851544627) +in the amount of client complexity for various use cases on the extra logic (some will +require more and others won't depending on the approach we pick). + +But the `dir` parameter does help us be a little more efficient on the homeserver. +Without `dir`, the homeserver has to check for a gap both forwards and backwards on +every request. If we include the `dir` parameter, we only have to check in one +direction. And then in both cases, the client still decides which direction they want +paginate further in. (this optimization is kind of a "who cares" point though) + +For the archive use case where we only care about showing the messages that happened on +a specific day, the `dir` parameter helps narrow down specifically on what we +specifically care about [without extra +logic](https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r851544627). +This applies to a similar use case for fetching windows of time (e.g. Jan 1 (dir = f) - +Jan 5 (dir = b)). + +For the messaging client use case, the user might care more about the closest event +regardless of direction. But since the closest event could be some unrenderable message +(like a reaction), the client still needs to find a suitable nearby event anyways making +the closest event a bit pointless. + + ### New `destination_server_ts` field Add a new field and index on messages called `destination_server_ts` which From b8d7ba3f93f53bf21243552f07525d592848a9cc Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Thu, 20 Oct 2022 16:46:46 -0500 Subject: [PATCH 26/28] Adjust to just suggest the right way See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r999099294 --- proposals/3030-jump-to-date.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index 3681b3fed73..aaed5bf5f1b 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -132,11 +132,15 @@ after that date is on January 27th." ### Abusing the `/timestamp_to_event` API to get the `m.room.create` event -Clients could abuse this new API for getting the `m.room.create` event, so -servers might want to put extra care into optimizing whatever lookups they do. -The create event contains quite a lot of information that a client needs in -order to operate, so it is frequently requested by said clients. For example, -the room type and room version (for displaying warnings about stability). +Although it's possible to jump to the start of the room and get the first event in the +room (`m.room.create`) with `/timestamp_to_event?dir=f&ts=0`, clients should still use +`GET /_matrix/client/v3/rooms/{roomId}/state/m.room.create/` to get the room creation +event. + +In the future, with things like importing history via +[MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716), the first +event you encounter with `/timestamp_to_event?dir=f&ts=0` could be an imported event before +the room was created. ## Alternatives From fabfb34bbc254a6e9cce3e48c4175629cb85574d Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 25 Oct 2022 14:23:58 -0500 Subject: [PATCH 27/28] =?UTF-8?q?Great=20simplification=20with=20the=20sam?= =?UTF-8?q?e=20meaning=20=F0=9F=8C=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/3030-jump-to-date.md | 46 +++++++++++++--------------------- 1 file changed, 17 insertions(+), 29 deletions(-) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index aaed5bf5f1b..f76c8472a71 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -197,35 +197,23 @@ also determines what part of the timeline to query. ### Return the closest event in any direction -The question you may be asking is "What's the rationale for including a `dir`? Can't we -just return the closest/nearest event regardless of direction?" Or at least add a -`dir=c` for the option to find the closest event. - -For clients, it doesn't make that much of a difference because you can always compare -the timestamp you made the request with to the `origin_server_ts` that was returned to -see what direction you need to paginate further in. And it's a -[toss-up](https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r851544627) -in the amount of client complexity for various use cases on the extra logic (some will -require more and others won't depending on the approach we pick). - -But the `dir` parameter does help us be a little more efficient on the homeserver. -Without `dir`, the homeserver has to check for a gap both forwards and backwards on -every request. If we include the `dir` parameter, we only have to check in one -direction. And then in both cases, the client still decides which direction they want -paginate further in. (this optimization is kind of a "who cares" point though) - -For the archive use case where we only care about showing the messages that happened on -a specific day, the `dir` parameter helps narrow down specifically on what we -specifically care about [without extra -logic](https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r851544627). -This applies to a similar use case for fetching windows of time (e.g. Jan 1 (dir = f) - -Jan 5 (dir = b)). - -For the messaging client use case, the user might care more about the closest event -regardless of direction. But since the closest event could be some unrenderable message -(like a reaction), the client still needs to find a suitable nearby event anyways making -the closest event a bit pointless. - +We considered omitting the `dir` parameter (or allowing `dir=c`) to have the server +return the closest event to the timestamp, regardless of direction. However, this seems +to offer little benefit. + +Firstly, for some usecases (such as archive viewing, where we want to show all the +messages that happened on a particular day), an explicit direction is important, so this +would have to be optional behaviour. + +For a regular messaging client, "directionless" search also offers little benefit: it is +easy for the client to repeat the request in the other direction if the returned event +is "too far away", and in any case it needs to manage an iterative search to handle +unrenderable events, as discussed above. + +Implementing a directionless search on the server carries a performance overhead, since +it must search both forwards and backwards on every request. In short, there is little +reason to expect that a single `dir=c` request would be any more efficient than a pair of +requests with `dir=b` and `dir=f`. ### New `destination_server_ts` field From 468a769ad777379553245e54232e1fcb0d94b53c Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Tue, 25 Oct 2022 14:34:33 -0500 Subject: [PATCH 28/28] Perfect is the enemy of good See https://github.com/matrix-org/matrix-spec-proposals/pull/3030#discussion_r1004651959 --- proposals/3030-jump-to-date.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proposals/3030-jump-to-date.md b/proposals/3030-jump-to-date.md index f76c8472a71..e19ce415b5f 100644 --- a/proposals/3030-jump-to-date.md +++ b/proposals/3030-jump-to-date.md @@ -101,11 +101,17 @@ token directly in another MSC ⏩ ### Receiving a rogue random delayed event ID +Since `origin_server_ts` is not enforcably accurate, we can only hope that an event's +`origin_server_ts` is relevant enough to its `prev_events` and descendants. + If you ask for "the message with `origin_server_ts` closest to Jan 1st 2018" you might actually get a rogue random delayed one that was backfilled from a federated server, but the human can figure that out by trying again with a slight variation on the date or something. +Since there isn't a good or fool-proof way to combat this, it's probably best to just go +with `origin_server_ts` and not let perfect be the enemy of good. + ### Receiving an unrenderable event ID