Implement user data import/export #3976

Nutomic · 2023-09-21T12:50:19Z

This adds new endpoints which lets users export their data (community follows, blocklists, profile settings), and import them again. It can be used to migrate between instances more easily, but also for backups in case the own instance goes down unexpectedly or the user gets banned. The user can create a new account on another instance, and import the latest backup.

dessalines

I'd rename this to UserSettingsBackup, because UserData seems like #506 , which means not just settings, but all user data, stuff like posts, comments, votes, saves, etc. Many of those things would be impossible to import, because its historical data.

Lots of somewhat overlapping issues this could close: https://github.com/LemmyNet/lemmy/issues?q=is%3Aissue+is%3Aopen+export

src/api_routes_http.rs

crates/db_schema/src/impls/local_user.rs

crates/apub/src/api/user_settings_backup.rs

wiki-me · 2023-09-28T09:36:55Z

backing up comments Could also be useful (e.g. if i want to refer to something i wrote before when writing a new comment), i realize this might overload the server but this seems like a classic case where you could offer paid features (and there is a strong economic justification for having the features paid), or just having instance admin manually approve it (or stop the process if needed).

Nutomic · 2023-09-28T09:52:06Z

src/api_routes_http.rs

@@ -291,7 +292,14 @@ pub fn config(cfg: &mut web::ServiceConfig, rate_limit: &RateLimitCell) {
          .route("/verify_email", web::post().to(verify_email))
          .route("/leave_admin", web::post().to(leave_admin))
          .route("/totp/generate", web::post().to(generate_totp_secret))
-          .route("/totp/update", web::post().to(update_totp)),
+          .route("/totp/update", web::post().to(update_totp))
+          .route("/export", web::get().to(export_user_backup)),


Should there be a stricter rate limit for export too?

Ya pry a good idea, the strictest we have.

Okay I applied the new import rate limit also for export (max once per day).

Nutomic · 2023-09-28T09:54:45Z

@wiki-me This is meant for data which can be exported from one account and then imported into another one. With comments thats not really possible, unless the import would repost all the comments again. That would result in countless duplicates. So if you want to backup everything you ever posted, it should be handled by an external tool using the API imo.

dessalines · 2023-09-28T12:34:12Z

@wiki-me That's the issue I linked above, #506 , which would export all account data, not just settings like this one does.

dessalines · 2023-09-28T14:00:54Z

src/api_routes_http.rs

-          .route(web::get().to(import_user_backup)),
+        web::scope("/user")
+          .wrap(rate_limit.import_user_settings())
+          .route("/export", web::get().to(export_user_backup))


Also, remember to rename these to export_settings, and the structs to UserSettingsBackup. In case in the future we want to export all user data, not just settings.

dessalines · 2023-09-29T14:05:36Z

src/api_routes_http.rs

@@ -297,8 +297,8 @@ pub fn config(cfg: &mut web::ServiceConfig, rate_limit: &RateLimitCell) {
      .service(
        web::scope("/user")
          .wrap(rate_limit.import_user_settings())
-          .route("/export", web::get().to(export_user_backup))
-          .route("/import", web::get().to(import_user_backup)),
+          .route("/export", web::get().to(export_settings))


The route urls should maybe change too, in case we add a user/export which includes everything.

/export_settings and import_settings

wiki-me · 2023-09-30T10:27:28Z

This is meant for data which can be exported from one account and then imported into another one. With comments thats not really possible, unless the import would repost all the comments again.

Could you just not send it to other servers, that way a user can checkout another user top posts (example given) , or controversal posts (if that is currently implemented for user profiles), in case you are considering if to invite him to some private community or doing some form of collaboration with him.

But i agree there are more important things to implement but maybe you want to "strike while the iron is hot".

kovalensky · 2023-10-01T13:20:16Z

Will user be able to export account settings after he/she was banned?

Nutomic · 2023-10-02T09:17:22Z

@kovalensky No, that would be a separate issue.

kovalensky · 2023-10-02T12:58:29Z

@Nutomic Is it necessary to open one?

Nutomic · 2023-10-02T14:32:44Z

@kovalensky Not sure, use the search function.

phiresky · 2023-10-04T14:00:24Z

crates/apub/src/api/user_settings_backup.rs

+
+  spawn_try_task(async move {
+    let person_id = local_user_view.person.id;
+    try_join_all(data.followed_communities.iter().map(|followed| async {


i don't think try_join_all is a good method to use here for two reasons:

if one of the follows fail, the others should still go through. for example if a user has followed communities on instances that no longer exist then the import will fail at a random point and leave the import partially done. (from the docs: If any future returns an error then all other futures will be canceled and an error will be returned immediately)

Instead, a method should be used where each error is handled separately with the main import keeping going.

There should be a limit on the concurrency. If the user follows 500 communities we don't want the server to fetch and subscribe to all of those 100 communities at once because rate limits might cause that to fail. It should either be fully sequential or with a fixed limit on concurrency.

This should be possible by converting the iterator into a stream and calling .buffered(10) (see https://stackoverflow.com/a/70871743/2639190 )

If these links are to be believed then join_all runs futures concurrently, not in parallel. Though I havent seen a clear confirmation for that in the docs.

https://stackoverflow.com/questions/63756169/rust-futuresfuturejoin-all-await-runs-futures-sequentially-but-parall

https://www.reddit.com/r/rust/comments/dt6u0s/joined_futures_will_run_concurrently_not_in/

If this is correct it would be enough to switch from try_join_all to join_all and collect the errors manually. I also tried to switch to the method you suggested, but its not working because the compiler throws an error about context geting moved.

futures::stream::iter(data.followed_communities.clone().into_iter().map( |followed| async move { // need to reset outgoing request count to avoid running into limit let context_ = context_.reset_request_count(); let community = followed.dereference(&context_).await?; let form = CommunityFollowerForm { person_id, community_id: community.id, pending: true, }; CommunityFollower::follow(&mut context_.pool(), &form).await?; LemmyResult::Ok(()) }, )) .buffer_unordered(10) .collect::<Vec<_>>() .await;

Okay I added error handling for remote fetches now. Also tried to return a list of failed items in the api response but couldnt get that working. For now its only being logged.

concurrently, not in parallel

Makes sense, but that'll still mean that it will send out 100 requests at almost the same time (since it'll mainly be waiting on the HTTP I/O), which will potentially cause issues

Got it working now. The code is rather verbose and repetetive now. It could potentially be simplified by moving the iter logic into a helper function, but that gets very complex with lots of generics and lambdas.

phiresky · 2023-10-11T14:28:09Z

migrations/2023-09-28-084231_import_user_settings_rate_limit/up.sql

+    ADD COLUMN import_user_settings int NOT NULL DEFAULT 1;
+
+ALTER TABLE local_site_rate_limit
+    ADD COLUMN import_user_settings_per_second int NOT NULL DEFAULT 86400;


i'm a bit confused by this naming. it sounds like it allows 86400 imports per second but what it really means is it allows only one import per 86400 seconds. so maybe it should be called smth like import_user_settings_rate_limit_seconds?

Yes all the rate limits have these confusing names so at least its consistent. It would be nice to rename them in the future to be clearer.

phiresky

lgtm

K4LCIFER · 2023-10-20T07:00:13Z

Has the Lemmy UI been given a button to allow the user to interact with this new feature yet, or is it just going to be through API interaction?

dessalines · 2023-10-20T14:17:54Z

Not yet no, you can track it over on that repo.

Implement endpoints for user data import/export

970af5a

Nutomic force-pushed the user-data-backup branch from 295a8c6 to 970af5a Compare September 21, 2023 12:51

dessalines reviewed Sep 26, 2023

View reviewed changes

src/api_routes_http.rs Outdated Show resolved Hide resolved

crates/db_schema/src/impls/local_user.rs Show resolved Hide resolved

add test

92f5651

Nutomic changed the title ~~Implement endpoints for user data import/export~~ Implement user data import/export Sep 27, 2023

Nutomic marked this pull request as ready for review September 27, 2023 15:02

Nutomic requested a review from phiresky as a code owner September 27, 2023 15:02

dessalines reviewed Sep 27, 2023

View reviewed changes

crates/apub/src/api/user_settings_backup.rs Outdated Show resolved Hide resolved

mormaer mentioned this pull request Sep 28, 2023

Account Syncing mlemgroup/mlem#663

Open

Nutomic added 3 commits September 28, 2023 10:36

exclude avatar/banner

20554f9

increase import url count, add rate limit

b3c7fed

also export/import saved posts

834c13a

Nutomic force-pushed the user-data-backup branch from cc12d14 to 834c13a Compare September 28, 2023 09:07

Merge branch 'main' into user-data-backup

97fbdce

Nutomic commented Sep 28, 2023

View reviewed changes

Nutomic mentioned this pull request Sep 28, 2023

Remove empty API responses #3993

Merged

rate limit

585a6df

dessalines reviewed Sep 28, 2023

View reviewed changes

rename

7993d9c

dessalines reviewed Sep 29, 2023

View reviewed changes

Nutomic mentioned this pull request Sep 29, 2023

Export & Import Community Subscriptons #3040

Closed

saved posts also exist

89f8bd2

This was referenced Sep 29, 2023

Import/Export Saved Posts & Comments #3918

Closed

Ability to share/export, and import user blocklists #3911

Closed

Moving user profile to a new instance #1985

Closed

mormaer mentioned this pull request Sep 30, 2023

[0.19.0] - Account backup option mlemgroup/mlem#675

Open

SrivatsanSenthilkumar mentioned this pull request Sep 30, 2023

Breaking changes to API in lemmy version 0.19 aeharding/voyager#745

Closed

rename routes

142002f

Nutomic mentioned this pull request Oct 2, 2023

Sync Users #3164

Closed

4 tasks

fix test

8173aac

Nutomic mentioned this pull request Oct 2, 2023

Implement a way for a user to export their account data. #506

Closed

phiresky reviewed Oct 4, 2023

View reviewed changes

dessalines approved these changes Oct 6, 2023

View reviewed changes

kovalensky mentioned this pull request Oct 6, 2023

Allow import/export of user data for banned accounts. #4025

Open

4 tasks

Nutomic and others added 5 commits October 9, 2023 11:40

error handling

12c85ce

Merge branch 'main' into user-data-backup

7828b32

clippy

6e5846e

limit parallelism

da02059

clippy

b8e704e

phiresky reviewed Oct 11, 2023

View reviewed changes

phiresky approved these changes Oct 11, 2023

View reviewed changes

Nutomic merged commit 6d7b38f into main Oct 11, 2023
2 checks passed

nodiscc mentioned this pull request Nov 18, 2023

Read and Saved posts CMahaff/lasim#8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement user data import/export #3976

Implement user data import/export #3976

Nutomic commented Sep 21, 2023

dessalines left a comment

wiki-me commented Sep 28, 2023

Nutomic Sep 28, 2023

dessalines Sep 28, 2023

Nutomic Sep 28, 2023

Nutomic commented Sep 28, 2023

dessalines commented Sep 28, 2023

dessalines Sep 28, 2023

Nutomic Sep 29, 2023

dessalines Sep 29, 2023

Nutomic Sep 29, 2023

dessalines Sep 29, 2023

Nutomic Oct 2, 2023

wiki-me commented Sep 30, 2023

kovalensky commented Oct 1, 2023

Nutomic commented Oct 2, 2023

kovalensky commented Oct 2, 2023

Nutomic commented Oct 2, 2023

phiresky Oct 4, 2023

Nutomic Oct 5, 2023

Nutomic Oct 9, 2023

phiresky Oct 10, 2023

Nutomic Oct 11, 2023

phiresky Oct 11, 2023

Nutomic Oct 11, 2023

phiresky left a comment

K4LCIFER commented Oct 20, 2023

dessalines commented Oct 20, 2023

Implement user data import/export #3976

Implement user data import/export #3976

Conversation

Nutomic commented Sep 21, 2023

dessalines left a comment

Choose a reason for hiding this comment

wiki-me commented Sep 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nutomic commented Sep 28, 2023

dessalines commented Sep 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wiki-me commented Sep 30, 2023

kovalensky commented Oct 1, 2023

Nutomic commented Oct 2, 2023

kovalensky commented Oct 2, 2023

Nutomic commented Oct 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phiresky left a comment

Choose a reason for hiding this comment

K4LCIFER commented Oct 20, 2023

dessalines commented Oct 20, 2023