Add account data to export command #14969

dklimpel · 2023-02-02T07:17:54Z

Lets export:

account data

Follows: #14894, same newsfile to consolidate

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Pull request includes a sign off
Code style is correct
(run the linters)

Signed-off-by: Dirk Klimpel 5740567+dklimpel@users.noreply.github.com

erikjohnston · 2023-02-14T09:40:19Z

synapse/app/admin_cmd.py

+        room_file = os.path.join(account_data_directory, room_id)
+
+        with open(room_file, "a") as f:
+            print(json.dumps(account_data), file=f)


This looks like its not used anymore?

Thanks! I have removed this.

clokep · 2023-02-14T12:33:23Z

Looks like this got birotted and needs develop merged in.

dklimpel · 2023-02-14T13:06:45Z

Looks like this got birotted and needs develop merged in.

Yes, there was some interactions with #14973

clokep · 2023-02-14T13:08:17Z

Yes, sorry about that! 😢

DMRobertson · 2023-02-16T18:13:36Z

synapse/app/admin_cmd.py

+
+        account_data_file = os.path.join(account_data_directory, file_name)
+
+        with open(account_data_file, "a") as f:


Why append mode here---is this what we typically do when exporting data?

As far as I can see there are some functions that need append, because e.g. room events are added. Here in this specific case not. Ican check in each function if append or write is needed, if it makes a difference. In any case the export starts with an empty directory.

Don't worry: I think it's okay as-is. I just wanted to understand what was going on (e.g. if we are expected to append to a previous data export).

DMRobertson · 2023-02-16T18:15:04Z

synapse/app/admin_cmd.py

+        account_data_file = os.path.join(account_data_directory, file_name)
+
+        with open(account_data_file, "a") as f:
+            print(json.dumps(account_data), file=f)


This is fine here (we don't expect account_data to be large), but for general interest it might be more efficient to use json.dump(account_data, f) (see here) for large account_data blobs.

Wouldn't that make sense for all functions in this file?

Probably! But we don't need to change it here and now.

clokep · 2023-02-16T19:08:24Z

docs/usage/administration/admin_faq.md

+    ├───account_data
+    │   ├───global
+    │   └───<room_id>


I wonder why not do rooms > account_data and then just have a user_data > account_data (or global_account_data)?

This has no special reasons. My thinking was that account data is user related rather than room related. I will change it.

Not anymore. The PR has now been merged.

It isn't a big deal, just curious what the thinking was!

They're related to both users & rooms, which is annoying in this case. 😄

@thezaidbintariq

Synapse 1.78.0 (2023-02-28) =========================== Bugfixes -------- - Fix a bug introduced in Synapse 1.76 where 5s delays would occasionally occur in deployments using workers. ([\#15150](matrix-org/synapse#15150)) Synapse 1.78.0rc1 (2023-02-21) ============================== Features -------- - Implement the experimental `exact_event_match` push rule condition from [MSC3758](matrix-org/matrix-spec-proposals#3758). ([\#14964](matrix-org/synapse#14964)) - Add account data to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.78/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#14969](matrix-org/synapse#14969)) - Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to disambiguate push rule keys with dots in them. ([\#15004](matrix-org/synapse#15004)) - Allow Synapse to use a specific Redis [logical database](https://redis.io/commands/select/) in worker-mode deployments. ([\#15034](matrix-org/synapse#15034)) - Tag opentracing spans for federation requests with the name of the worker serving the request. ([\#15042](matrix-org/synapse#15042)) - Implement the experimental `exact_event_property_contains` push rule condition from [MSC3966](matrix-org/matrix-spec-proposals#3966). ([\#15045](matrix-org/synapse#15045)) - Remove spurious `dont_notify` action from the defaults for the `.m.rule.reaction` pushrule. ([\#15073](matrix-org/synapse#15073)) - Update the error code returned when user sends a duplicate annotation. ([\#15075](matrix-org/synapse#15075)) Bugfixes -------- - Prevent clients from reporting nonexistent events. ([\#13779](matrix-org/synapse#13779)) - Return spec-compliant JSON errors when unknown endpoints are requested. ([\#14605](matrix-org/synapse#14605)) - Fix a long-standing bug where the room aliases returned could be corrupted. ([\#15038](matrix-org/synapse#15038)) - Fix a bug introduced in Synapse 1.76.0 where partially-joined rooms could not be deleted using the [purge room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api). ([\#15068](matrix-org/synapse#15068)) - Fix a long-standing bug where federated joins would fail if the first server in the list of servers to try is not in the room. ([\#15074](matrix-org/synapse#15074)) - Fix a bug introduced in Synapse v1.74.0 where searching with colons when using ICU for search term tokenisation would fail with an error. ([\#15079](matrix-org/synapse#15079)) - Reduce the likelihood of a rare race condition where rejoining a restricted room over federation would fail. ([\#15080](matrix-org/synapse#15080)) - Fix a bug introduced in Synapse 1.76 where workers would fail to start if the `health` listener was configured. ([\#15096](matrix-org/synapse#15096)) - Fix a bug introduced in Synapse 1.75 where the [portdb script](https://matrix-org.github.io/synapse/release-v1.78/postgres.html#porting-from-sqlite) would fail to run after a room had been faster-joined. ([\#15108](matrix-org/synapse#15108)) Improved Documentation ---------------------- - Document how to start Synapse with Poetry. Contributed by @thezaidbintariq. ([\#14892](matrix-org/synapse#14892), [\#15022](matrix-org/synapse#15022)) - Update delegation documentation to clarify that SRV DNS delegation does not eliminate all needs to serve files from .well-known locations. Contributed by @williamkray. ([\#14959](matrix-org/synapse#14959)) - Fix a mistake in registration_shared_secret_path docs. ([\#15078](matrix-org/synapse#15078)) - Refer to a more recent blog post on the [Database Maintenance Tools](https://matrix-org.github.io/synapse/latest/usage/administration/database_maintenance_tools.html) page. Contributed by @jahway603. ([\#15083](matrix-org/synapse#15083)) Internal Changes ---------------- - Re-type hint some collections as read-only. ([\#13755](matrix-org/synapse#13755)) - Faster joins: don't stall when another user joins during a partial-state room resync. ([\#14606](matrix-org/synapse#14606)) - Add a class `UnpersistedEventContext` to allow for the batching up of storing state groups. ([\#14675](matrix-org/synapse#14675)) - Add a check to ensure that locked dependencies have source distributions available. ([\#14742](matrix-org/synapse#14742)) - Tweak comment on `_is_local_room_accessible` as part of room visibility in `/hierarchy` to clarify the condition for a room being visible. ([\#14834](matrix-org/synapse#14834)) - Prevent `WARNING: there is already a transaction in progress` lines appearing in PostgreSQL's logs on some occasions. ([\#14840](matrix-org/synapse#14840)) - Use `StrCollection` to avoid potential bugs with `Collection[str]`. ([\#14929](matrix-org/synapse#14929)) - Improve performance of `/sync` in a few situations. ([\#14973](matrix-org/synapse#14973)) - Limit concurrent event creation for a room to avoid state resolution when sending bursts of events to a local room. ([\#14977](matrix-org/synapse#14977)) - Skip calculating unread push actions in /sync when enable_push is false. ([\#14980](matrix-org/synapse#14980)) - Add a schema dump symlinks inside `contrib`, to make it easier for IDEs to interrogate Synapse's database schema. ([\#14982](matrix-org/synapse#14982)) - Improve type hints. ([\#15008](matrix-org/synapse#15008), [\#15026](matrix-org/synapse#15026), [\#15027](matrix-org/synapse#15027), [\#15028](matrix-org/synapse#15028), [\#15031](matrix-org/synapse#15031), [\#15035](matrix-org/synapse#15035), [\#15052](matrix-org/synapse#15052), [\#15072](matrix-org/synapse#15072), [\#15084](matrix-org/synapse#15084)) - Update [MSC3952](matrix-org/matrix-spec-proposals#3952) support based on changes to the MSC. ([\#15037](matrix-org/synapse#15037)) - Avoid mutating a cached value in `get_user_devices_from_cache`. ([\#15040](matrix-org/synapse#15040)) - Fix a rare exception in logs on start up. ([\#15041](matrix-org/synapse#15041)) - Update pyo3-log to v0.8.1. ([\#15043](matrix-org/synapse#15043)) - Avoid mutating cached values in `_generate_sync_entry_for_account_data`. ([\#15047](matrix-org/synapse#15047)) - Refactor arguments of `try_unbind_threepid` and `_try_unbind_threepid_with_id_server` to not use dictionaries. ([\#15053](matrix-org/synapse#15053)) - Merge debug logging from the hotfixes branch. ([\#15054](matrix-org/synapse#15054)) - Faster joins: omit device list updates originating from partial state rooms in /sync responses without lazy loading of members enabled. ([\#15069](matrix-org/synapse#15069)) - Fix clashing database transaction name. ([\#15070](matrix-org/synapse#15070)) - Upper-bound frozendict dependency. This works around us being unable to test installing our wheels against Python 3.11 in CI. ([\#15114](matrix-org/synapse#15114)) - Tweak logging for when a worker waits for its view of a replication stream to catch up. ([\#15120](matrix-org/synapse#15120)) <details><summary>Locked dependency updates</summary> - Bump bleach from 5.0.1 to 6.0.0. ([\#15059](matrix-org/synapse#15059)) - Bump cryptography from 38.0.4 to 39.0.1. ([\#15020](matrix-org/synapse#15020)) - Bump ruff version from 0.0.230 to 0.0.237. ([\#15033](matrix-org/synapse#15033)) - Bump dtolnay/rust-toolchain from 9cd00a88a73addc8617065438eff914dd08d0955 to 25dc93b901a87e864900a8aec6c12e9aa794c0c3. ([\#15060](matrix-org/synapse#15060)) - Bump systemd-python from 234 to 235. ([\#15061](matrix-org/synapse#15061)) - Bump serde_json from 1.0.92 to 1.0.93. ([\#15062](matrix-org/synapse#15062)) - Bump types-requests from 2.28.11.8 to 2.28.11.12. ([\#15063](matrix-org/synapse#15063)) - Bump types-pillow from 9.4.0.5 to 9.4.0.10. ([\#15064](matrix-org/synapse#15064)) - Bump sentry-sdk from 1.13.0 to 1.15.0. ([\#15065](matrix-org/synapse#15065)) - Bump types-jsonschema from 4.17.0.3 to 4.17.0.5. ([\#15099](matrix-org/synapse#15099)) - Bump types-bleach from 5.0.3.1 to 6.0.0.0. ([\#15100](matrix-org/synapse#15100)) - Bump dtolnay/rust-toolchain from 25dc93b901a87e864900a8aec6c12e9aa794c0c3 to e12eda571dc9a5ee5d58eecf4738ec291c66f295. ([\#15101](matrix-org/synapse#15101)) - Bump dawidd6/action-download-artifact from 2.24.3 to 2.25.0. ([\#15102](matrix-org/synapse#15102)) - Bump types-pillow from 9.4.0.10 to 9.4.0.13. ([\#15104](matrix-org/synapse#15104)) - Bump types-setuptools from 67.1.0.0 to 67.3.0.1. ([\#15105](matrix-org/synapse#15105)) </details>

Add account data to to export command

b8fca46

dklimpel requested a review from a team as a code owner February 2, 2023 07:17

github-actions bot deployed to PR Documentation Preview February 2, 2023 07:18 View deployment

newsfile

9811c51

github-actions bot deployed to PR Documentation Preview February 2, 2023 07:20 View deployment

erikjohnston approved these changes Feb 14, 2023

View reviewed changes

Merge branch 'develop' into more_export

bce4861

github-actions bot deployed to PR Documentation Preview February 14, 2023 11:07 View deployment

dklimpel added 2 commits February 14, 2023 12:08

remove not needed function

ab2b030

update newsfile

16e7495

github-actions bot deployed to PR Documentation Preview February 14, 2023 11:11 View deployment

adopt matrix-org#14973

6dfbf5a

github-actions bot deployed to PR Documentation Preview February 14, 2023 12:59 View deployment

dklimpel requested a review from erikjohnston February 14, 2023 13:04

DMRobertson approved these changes Feb 16, 2023

View reviewed changes

DMRobertson changed the title ~~Add account data to to export command~~ Add account data to export command Feb 16, 2023

clokep reviewed Feb 16, 2023

View reviewed changes

DMRobertson merged commit 61bfcd6 into matrix-org:develop Feb 17, 2023

dklimpel deleted the more_export branch February 17, 2023 15:17

This was referenced Feb 18, 2023

Use json.dump in FileExfiltrationWriter #15095

Merged

Add media information to export command #15107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add account data to export command #14969

Add account data to export command #14969

dklimpel commented Feb 2, 2023 •

edited

erikjohnston Feb 14, 2023

dklimpel Feb 14, 2023

clokep commented Feb 14, 2023

dklimpel commented Feb 14, 2023

clokep commented Feb 14, 2023

DMRobertson Feb 16, 2023

dklimpel Feb 17, 2023

DMRobertson Feb 17, 2023

DMRobertson Feb 16, 2023

dklimpel Feb 17, 2023

DMRobertson Feb 17, 2023

clokep Feb 16, 2023

dklimpel Feb 17, 2023

dklimpel Feb 17, 2023

clokep Feb 17, 2023


		account_data_file = os.path.join(account_data_directory, file_name)

		with open(account_data_file, "a") as f:

Add account data to export command #14969

Add account data to export command #14969

Conversation

dklimpel commented Feb 2, 2023 • edited

Pull Request Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Feb 14, 2023

dklimpel commented Feb 14, 2023

clokep commented Feb 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dklimpel commented Feb 2, 2023 •

edited