WIP: Allow users to export their messages from the UI #8834

punchagan · 2018-03-27T04:14:50Z

Testing Plan:

GIFs or Screenshots:

timabbott · 2018-03-27T20:52:35Z

static/js/settings_account.js

+        e.preventDefault();
+        e.stopPropagation();
+        channel.get({
+            url: "/json/users/me/export-my-data",


I think /export would be a more RESTful UI.

timabbott · 2018-03-27T20:52:54Z

static/js/settings_account.js

+                $('#user_export_data_message').text(response.msg).addClass('show');
+            },
+            error: function(response){
+                // FIXME: Does error have response.msg??!


yep, it does. But see the ui_report library.

timabbott · 2018-03-27T20:53:25Z

static/templates/settings/account-settings.handlebars

+                    </button>
+                </div>
+            </form>
+        </div>


I think we might want to move this into the below section and title it was "Account options."

timabbott · 2018-03-27T20:53:38Z

static/templates/settings/account-settings.handlebars

+            <form class="grid">
+                <div class="input-group">
+                    <button type="submit" class="button rounded w-200 input-size" id="user_export_data">
+                        {{t 'Export my messages' }}


Maybe "Export my Zulip message history"?

timabbott · 2018-03-27T20:54:22Z

zerver/views/user_settings.py

@@ -229,3 +230,13 @@ def change_enter_sends(request: HttpRequest, user_profile: UserProfile,
                       enter_sends: bool=REQ(validator=check_bool)) -> HttpResponse:
    do_change_enter_sends(user_profile, enter_sends)
    return json_success()
+
+# FIXME: Do we plan to allow exports of all messages sent by bots?


I think human_users_only is reasonable for this.

timabbott · 2018-03-27T20:54:40Z

zerver/worker/queue_processors.py

+        user_profile = get_user_profile_by_email(email)
+
+        # FIXME: Can we check if an export is currently running, or has been
+        # started in the past hour, and deny exports?


We'll want to do something here.

Probably should just manage this with a new UserDataExport table.

timabbott · 2018-03-27T20:54:57Z

zerver/worker/queue_processors.py

@@ -527,3 +535,37 @@ def consume(self, event: Mapping[str, Any]) -> None:
                (stream, recipient, sub) = access_stream_by_id(user_profile, stream_id,
                                                               require_active=False)
                do_mark_stream_messages_as_read(user_profile, stream)
+
+
+@assign_queue('export_data')


Let's use the deferred_work queue; this seems like a rare enough thing that we don't need several.

timabbott · 2018-03-27T20:55:39Z

zproject/urls.py

@@ -258,6 +258,10 @@
    url(r'^users/me/avatar$', rest_dispatch,
        {'POST': 'zerver.views.user_settings.set_avatar_backend',
         'DELETE': 'zerver.views.user_settings.delete_avatar_backend'}),
+    url(r'^users/me/export-my-data$', rest_dispatch,
+        # FIXME: Use a POST?
+        {'GET': 'zerver.views.user_settings.export_my_data'}),


Yes, this should be a POST; we don't want browsers to auto-retry this sort of request.

timabbott · 2018-03-27T20:57:31Z

zerver/worker/queue_processors.py

+            size = getsize(tarball_path)
+            name = basename(tarball_path)
+            print(content_type, name, size)
+            upload_path = upload_message_image(name, size, content_type, f.read(), user_profile)


Hmm, is that function really called upload_message_image? Sigh, that name is super confusing. Can you do a separate PR (or commit at the base of this PR) to rename that to upload_message_file? That seems like a clear thing to merge first. Be sure to use good commit discipline for that refactor; check out the Zulip commit message guidelines for more details: https://zulip.readthedocs.io/en/latest/contributing/version-control.html

timabbott · 2018-03-27T20:58:55Z

zerver/worker/queue_processors.py

+    def consume(self, email: str) -> None:
+        output_dir = tempfile.mkdtemp(prefix="/tmp/zulip-export-")
+        if os.path.exists(output_dir):
+            shutil.rmtree(output_dir)


We can probably just assert that it doesn't exist; that should never happen given that mkdtemp is supposed to make us a secure, randomly generated path. Also, does mkdtemp create the directory, in which case we should be asserting it does exist?

Yes, this code is something I just copied over from the export command, and your suggestions make sense. 👍

timabbott · 2018-03-27T20:59:24Z

zerver/worker/queue_processors.py

+        print("Finished exporting to %s; tarring" % (output_dir,))
+        tarball_path = output_dir.rstrip('/') + '.tar.gz'
+        subprocess.check_call(["tar", "--strip-components=1", "-czf", tarball_path, output_dir])
+        print("Tarball written to %s" % (tarball_path,))


We should be sure to delete the file after uploading.

punchagan · 2018-03-29T05:15:56Z

@timabbott I pushed a few changes based on your initial feedback. I also implemented some tracking for a running export, but I'm not sure what the best way to handle failures during an export would be.

Also, like you said, the export part could definitely use some more tests and error handling. Any advice on that would be helpful too.

timabbott · 2018-03-30T20:46:00Z

zerver/worker/queue_processors.py

+        super(DeferredWorker, self).consume_wrapper(data)
+        if data['type'] == 'export_user_messages':
+            user_id = data['user_profile_id']
+            UserDataExport.export_finished(user_id)


This might be easier done as a try/except block in the handler after extracting the function.

punchagan · 2018-07-12T09:05:32Z

@timabbott I've finally been able to address the suggestion to update the status of the export for the user using an event. This PR is ready for another review.

zulipbot · 2018-07-24T00:42:49Z

Heads up @punchagan, we just merged some commits that conflict with the changes your made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the upstream/master branch and resolve your pull request's merge conflicts accordingly.

brainwane · 2020-03-30T19:27:11Z

@punchagan It looks like Tim made several other suggestions/requests in his review - did you address those already?

It would be great to get this feature! :-)

punchagan · 2020-03-31T07:26:08Z

Hey @brainwane thanks for the nudge/ping on this. This PR has become quite stale, and I will try to take a look at this again. (I seem to have some time on my hands owing to the pandemic situation).

Meanwhile, this PR also doesn't really implement the exact feature you requested - dump of all the messages you sent. This PR tries to create an export of all sent and received messages. I remember @timabbott mentioned on CZO that just getting the messages you sent would be easy to do using the API. Would it be useful to look into that and write a script that fetches this data, before I jump back into this PR? 🤔

zulipbot added the size: L label Mar 27, 2018

punchagan force-pushed the issue/2047-export-my-data branch 2 times, most recently from 6b60e18 to d7819d9 Compare March 27, 2018 17:07

timabbott reviewed Mar 27, 2018

View reviewed changes

punchagan force-pushed the issue/2047-export-my-data branch 4 times, most recently from 5db4cf6 to ecb8fbc Compare March 28, 2018 17:11

zulipbot added size: XL and removed size: L labels Mar 28, 2018

punchagan force-pushed the issue/2047-export-my-data branch 3 times, most recently from e796050 to 9d8f28a Compare March 29, 2018 05:06

punchagan force-pushed the issue/2047-export-my-data branch 6 times, most recently from 49a5294 to 15985bb Compare March 30, 2018 00:38

timabbott reviewed Mar 30, 2018

View reviewed changes

punchagan force-pushed the issue/2047-export-my-data branch 4 times, most recently from 2fe5990 to 90cf68f Compare April 2, 2018 05:12

punchagan force-pushed the issue/2047-export-my-data branch 4 times, most recently from 04ffd11 to 1ebb0a7 Compare May 3, 2018 05:20

adnrs96 mentioned this pull request May 11, 2018

Migration: Move from the use of typing.Text to use str #9360

Closed

punchagan force-pushed the issue/2047-export-my-data branch 4 times, most recently from dc923a9 to 227f310 Compare July 12, 2018 07:39

punchagan added 6 commits July 12, 2018 13:35

settings: Allow specifying saving and failure message in opts

5dbe0b1

export: Add end-point to allow exporting of user's own data

f1f32e9

export: Ensure a single export runs per user at any given time

3cd4bee

export: Add a button in the settings to start an export

bab2198

export: Extract tarball creation of export dir into a function

581ca8f

upload: New upload function for file on disk

f7cdac2

punchagan force-pushed the issue/2047-export-my-data branch from 227f310 to f7cdac2 Compare July 12, 2018 08:06

punchagan mentioned this pull request Jul 14, 2018

I own my own words: one-click export of my own messages #2047

Open

zulipbot added the has conflicts label Aug 7, 2018

seanbreckenridge mentioned this pull request Jan 27, 2022

initial my.zulip module karlicoss/HPI#197

Merged

timabbott force-pushed the main branch 2 times, most recently from 4ec3636 to 88b200c Compare August 18, 2023 23:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Allow users to export their messages from the UI #8834

WIP: Allow users to export their messages from the UI #8834

punchagan commented Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

timabbott Mar 27, 2018

punchagan Mar 28, 2018

timabbott Mar 27, 2018

punchagan commented Mar 29, 2018

timabbott Mar 30, 2018

punchagan commented Jul 12, 2018

zulipbot commented Jul 24, 2018

brainwane commented Mar 30, 2020

punchagan commented Mar 31, 2020

WIP: Allow users to export their messages from the UI #8834

Are you sure you want to change the base?

WIP: Allow users to export their messages from the UI #8834

Conversation

punchagan commented Mar 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

punchagan commented Mar 29, 2018

Choose a reason for hiding this comment

punchagan commented Jul 12, 2018

zulipbot commented Jul 24, 2018

brainwane commented Mar 30, 2020

punchagan commented Mar 31, 2020