Implement MSC3383: include destination in X-Matrix auth header #11398

jcgruenhage · 2021-11-19T14:07:11Z

This is the implementation for MSC3383, as requested over at matrix-org/matrix-spec-proposals#3383.
We've tested this at Famedly already, deployed it to a testserver, federation still works as expected, we've tested it with complement and sytest, and we're planning to roll it out with the 1.47.1 release of our synapse fork next week.

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
Pull request includes a sign off (see 9b74330)
Code style is correct
(run the linters)

DMRobertson

A few thoughts! Probably wants more input from @matrix-org/synapse-core.

synapse/federation/transport/server/_base.py

synapse/http/matrixfederationclient.py

clokep

I think this looks great overall, but we would need to see the MSC move forward before merging.

We might also be able to include it if it were behind an experimental configuration flag, but I'm unsure in this case.

changelog.d/11398.feature

ShadowJonathan · 2021-11-23T12:59:46Z

but we would need to see the MSC move forward before merging.

That'd be a bit of a catch-22, as a MSC needs a working implementation before it usually is seriously considered for SCT review (at least, that's the defacto modus operandi), I only see approvals, so unless you'd like to review it as well (and see it move forward yourself), then i think this has to go forward first.

Unless you'd like to see this implemented in an alternative server implementation as well...?

clokep · 2021-11-23T13:10:34Z

as a MSC needs a working implementation before it usually is seriously considered for SCT review

A pull request is considered a working implementation.

clokep · 2021-11-23T13:11:04Z

@jcgruenhage Looks like this got caught in the linter though, Could you re-run ./scripts-dev/lint.sh?

jcgruenhage · 2021-11-23T15:25:34Z

Yeah, sorry, I'll run the linting stuff later, didn't have time to clean that up yesterday.

matrix-org/matrix-spec-proposals#3383 Co-Authored-By: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>

jcgruenhage · 2021-11-27T11:52:12Z

Linting stuff is all fixed by now, and I also changed the parsing of the header a bit, because the linter complained on that as well, even though that line hadn't really been touched in 3 years afaict

clokep · 2022-04-19T11:21:45Z

The MSC has passed FCP, so putting this back into the review queue.

@jcgruenhage might be good to merge develop into this since it has been a while since CI ran?

jcgruenhage · 2022-04-19T12:35:32Z

Merging develop as opposed to rebasing this? Not that much of a fan of merging long diverged branches, but can do if that's your process.

changelog.d/11398.feature

synapse/federation/transport/server/_base.py

clokep · 2022-04-19T12:38:07Z

Merging develop as opposed to rebasing this? Not that much of a fan of merging long diverged branches, but can do if that's your process.

We prefer merges -- there's some info in the docs about why: https://matrix-org.github.io/synapse/latest/development/git.html

Note that we squash merge so it will end up being a single commit once merged to develop.

jcgruenhage · 2022-04-19T13:14:27Z

Okay, sounds good.

Conflicts: synapse/federation/transport/server/_base.py

…cted destination and actual destination don't match up

DMRobertson

I think this needs the linter script to get it through CI; otherwise, I'm happy.

DMRobertson

Many thanks!

@Bubu

Synapse 1.58.0rc1 (2022-04-26) ============================== As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61. Features -------- - Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398)) - Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537)) - Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465)) - Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir. ([\#12427](matrix-org/synapse#12427)) - Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543)) Bugfixes -------- - Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213)) - Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319)) - Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476)) - Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496)) - Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510)) - Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520)) - Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522)) Improved Documentation ---------------------- - Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340)) - Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527)) - Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451)) - Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457)) - Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475)) - Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492)) - Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495)) - Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501)) - Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533)) Deprecations and Removals ------------------------- - The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344)) - Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382)) Internal Changes ---------------- - Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394)) - Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399)) - Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395)) - Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472)) - Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445)) - Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450)) - Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497)) - Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455)) - Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454)) - Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464)) - Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466)) - complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467)) - Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474)) - Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483)) - Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511)) - Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519)) - Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528)) - Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468)) - Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))

@Bubu

Synapse 1.58.0 (2022-05-03) =========================== As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61. No significant changes since 1.58.0rc2. Synapse 1.58.0rc2 (2022-04-26) ============================== This release candidate fixes bugs related to Synapse 1.58.0rc1's logic for handling device list updates. Bugfixes -------- - Fix a bug introduced in Synapse 1.58.0rc1 where the main process could consume excessive amounts of CPU and memory while handling sentry logging failures. ([\#12554](matrix-org/synapse#12554)) - Fix a bug introduced in Synapse 1.58.0rc1 where opentracing contexts were not correctly sent to whitelisted remote servers with device lists updates. ([\#12555](matrix-org/synapse#12555)) Internal Changes ---------------- - Reduce unnecessary work when handling remote device list updates. ([\#12557](matrix-org/synapse#12557)) Synapse 1.58.0rc1 (2022-04-26) ============================== Features -------- - Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398)) - Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537)) - Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465)) - Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir @ Beeper. ([\#12427](matrix-org/synapse#12427)) - Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543)) Bugfixes -------- - Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213)) - Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319)) - Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476)) - Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496)) - Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510)) - Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520)) - Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522)) Improved Documentation ---------------------- - Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340)) - Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527)) - Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451)) - Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457)) - Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475)) - Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492)) - Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495)) - Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501)) - Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533)) Deprecations and Removals ------------------------- - The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344)) - Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382)) Internal Changes ---------------- - Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394)) - Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399)) - Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395)) - Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472)) - Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445)) - Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450)) - Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497)) - Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455)) - Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454)) - Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464)) - Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466)) - complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467)) - Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474)) - Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483)) - Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511)) - Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519)) - Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528)) - Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468)) - Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))

@Bubu

Change to wheel/poetry from egg. Port remediation of upstream's cryptography version demands to new build system. Upstream no longer installs synmark. Upstream NEWS, less bugfixes and minor updates: Synapse 1.58.1 (2022-05-05) =========================== [Debian packaging bugfix] Synapse 1.58.0 (2022-05-03) =========================== As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61. Synapse 1.58.0rc1 (2022-04-26) ============================== Features -------- - Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398)) - Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537)) - Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465)) - Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir @ Beeper. ([\#12427](matrix-org/synapse#12427)) - Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543)) Improved Documentation ---------------------- - Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475)) Deprecations and Removals ------------------------- - The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344)) - Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382))

jcgruenhage requested a review from a team as a code owner November 19, 2021 14:07

jcgruenhage force-pushed the jcgruenhage/msc3383 branch 2 times, most recently from efa6847 to 5f0c507 Compare November 19, 2021 14:21

DMRobertson self-assigned this Nov 22, 2021

DMRobertson reviewed Nov 22, 2021

View reviewed changes

synapse/federation/transport/server/_base.py Outdated Show resolved Hide resolved

synapse/federation/transport/server/_base.py Outdated Show resolved Hide resolved

synapse/http/matrixfederationclient.py Show resolved Hide resolved

DMRobertson requested a review from a team November 22, 2021 12:26

DMRobertson removed their assignment Nov 22, 2021

jcgruenhage force-pushed the jcgruenhage/msc3383 branch 4 times, most recently from f45c6d8 to b8e413b Compare November 22, 2021 13:16

clokep reviewed Nov 23, 2021

View reviewed changes

changelog.d/11398.feature Outdated Show resolved Hide resolved

DMRobertson added the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Nov 26, 2021

Bubu and others added 4 commits November 27, 2021 12:18

MSC3383: include destination in X-Matrix auth header

9b74330

matrix-org/matrix-spec-proposals#3383 Co-Authored-By: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>

use HTTPStatus constants in federation base transport

2cb31e8

set type hints for auth header parsing method

8c8d84e

add type annotations to federation auth header parsing

69f606c

jcgruenhage force-pushed the jcgruenhage/msc3383 branch from b8e413b to 69f606c Compare November 27, 2021 11:18

clokep removed the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Nov 29, 2021

clokep changed the title ~~MSC3383: include destination in X-Matrix auth header~~ Implement MSC3383: include destination in X-Matrix auth header Nov 29, 2021

clokep added the z-blocked (Deprecated Label) label Nov 29, 2021

uhoreg mentioned this pull request Jan 10, 2022

MSC3383: Include destination in X-Matrix Auth Header matrix-org/matrix-spec-proposals#3383

Merged

clokep requested a review from a team April 19, 2022 11:21

DMRobertson reviewed Apr 19, 2022

View reviewed changes

changelog.d/11398.feature Outdated Show resolved Hide resolved

synapse/federation/transport/server/_base.py Outdated Show resolved Hide resolved

jcgruenhage added 3 commits April 19, 2022 15:37

Merge branch 'develop'

658e4ab

Conflicts: synapse/federation/transport/server/_base.py

avoid redirect in changelog item

ad2d91c

address note about error code mentioned in the MSC, for when the expe…

a807f8a

…cted destination and actual destination don't match up

DMRobertson suggested changes Apr 19, 2022

View reviewed changes

fix linting issues

1d73c70

jcgruenhage requested a review from DMRobertson April 19, 2022 15:07

DMRobertson removed the z-blocked (Deprecated Label) label Apr 19, 2022

DMRobertson approved these changes Apr 19, 2022

View reviewed changes

DMRobertson merged commit a1f87f5 into matrix-org:develop Apr 19, 2022

DMRobertson mentioned this pull request Jun 14, 2022

Fix destination_is errors seen in sentry. #13041

Merged

clokep mentioned this pull request Sep 29, 2022

Support Matrix 1.3 #13953

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement MSC3383: include destination in X-Matrix auth header #11398

Implement MSC3383: include destination in X-Matrix auth header #11398

jcgruenhage commented Nov 19, 2021 •

edited

DMRobertson left a comment

clokep left a comment

ShadowJonathan commented Nov 23, 2021 •

edited

clokep commented Nov 23, 2021

clokep commented Nov 23, 2021

jcgruenhage commented Nov 23, 2021 via email

jcgruenhage commented Nov 27, 2021

clokep commented Apr 19, 2022

jcgruenhage commented Apr 19, 2022

clokep commented Apr 19, 2022

jcgruenhage commented Apr 19, 2022

DMRobertson left a comment

DMRobertson left a comment

Implement MSC3383: include destination in X-Matrix auth header #11398

Implement MSC3383: include destination in X-Matrix auth header #11398

Conversation

jcgruenhage commented Nov 19, 2021 • edited

Pull Request Checklist

DMRobertson left a comment

Choose a reason for hiding this comment

clokep left a comment

Choose a reason for hiding this comment

ShadowJonathan commented Nov 23, 2021 • edited

clokep commented Nov 23, 2021

clokep commented Nov 23, 2021

jcgruenhage commented Nov 23, 2021 via email

jcgruenhage commented Nov 27, 2021

clokep commented Apr 19, 2022

jcgruenhage commented Apr 19, 2022

clokep commented Apr 19, 2022

jcgruenhage commented Apr 19, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

DMRobertson left a comment

Choose a reason for hiding this comment

jcgruenhage commented Nov 19, 2021 •

edited

ShadowJonathan commented Nov 23, 2021 •

edited