export docs and related docs #7

ETLaurent · 2023-08-15T10:16:40Z

Summary

Depends on make changes for import/export module to work properly apostrophe#4270
Read PR comments, TODOs and FIXMEs

Big choucroute to batch-export documents, export a single document and page.

Export their actual data that is in the database (we don't use the manager that is populating fields and adding data).
We export the docs as they are in the DB.

We only use the docs manager to get the related docs.

Then we compress the docs in a aposDocs.json that's inside a ZIP or GZIP archive, and we return the URL to the client that's opened by the browser for download.

What are the specific steps to test this change?

single doc (you can also test with a page):

multiple docs:

What kind of change does this PR introduce?

(Check at least one)

Make sure the PR fulfills these requirements:

It includes a) the existing issue ID being resolved, b) a convincing reason for adding this feature, or c) a clear description of the bug it resolves
The changelog is updated
Related documentation has been updated
Related tests have been updated

If adding a new feature without an already open issue, it's best to open a feature request issue first and wait for approval before working on it.

Other information:

linear · 2023-08-15T10:16:42Z

PRO-4261 As an editor, I can download a ZIP file of my exported documents

When clicking on the download button of the export modal, after I selected the related docs that I want to include in the export, the process must be triggered.

image.png

Two new routes export must exist in the modules piece-type and page. Both routes will call a method export from the module doc-type .

This method should take the IDs of the documents the user want to export, as well as the related types that he wants to get. Only the types, not the IDs, since we decided that it would be too heavy for the export modal to get the amount of related docs when we batch a large amount of documents.

Generate files

This method will need to use the relate method from the module schema, this method fills documents with related ones.

Since we need to store documents in a flat EJSON array, it might be a good idea to add an option to this method in order to get a flat array of related documents instead of filling requested documents with their related docs, see how it could potentially be used with query builders.

We want one EJSON file for each db collection, for example aposDocs.json and aposAttachment.json, containing arrays of documents..

We want, for each document to export the draft and live versions (only draft if there is no live). The draft version must appear first to avoid situation where a live exists without draft which is forbidden.

Same thing for related documents, they must be written first in the EJSON file, otherwise the doc manager will yell when we'll import it.

We will create an attachment folder when actual attachments will be stored (images, files…).

Generate Zip

To generate the ZIP file we would prefer using the native node library.

Progress bar notification

This progress bar notification is triggered by wrapping your export function with the run method of the job module. Here is an example from the piece-type-importer module:

return self.apos.modules['apostrophecms/job'].run(
  req,
  (req, reporting, { notificationId }) => self.importRun(req, {
    progressNotifId: notificationId,
    filePath: file.path,
    reporting,
    totalPieces
  }),
  {}
);

The reporting object passed from the second argument function allows to set the total of the documents you want to import with the method setTotal, as well as to call the methods success or failure for every document that we tried to export or import.

Finally the ZIP file will be sent back to the client and the download triggered from the user's browser.

Permissions

For exporting documents, a user that has at least one per-type permission should be able to export anything. He won’t be able to see adminOnly documents, therefore should not be able to export these documents. It matches the way it works currently because in this case the user can see all drafts. In core, admin should be able to export adminOnly content, except for users.

For Reference

https://github.com/apostrophecms/tech-designs/blob/main/3.x/share-documents-across-sites/design.md

Acceptance Criteria

I'm an editor or admin on with the Download modal open, and my documents selected, and I hit "Download" and when I do, a ZIP file begins downloading.
The zip file downloads and I open it and I see the the draft and live versions (only draft if there is no live) in my unarchived folder.

ETLaurent · 2023-08-17T11:40:46Z

index.js

+    }
+    if (self.options.import !== false) {
+      self.apos.asset.iconMap['apos-import-export-upload-icon'] = 'Upload';
+    }


not related to this PR, but improves this code

lib/methods.js

ETLaurent · 2023-08-23T16:25:46Z

lib/methods/export.js

+      const data = {
+        'aposDocs.json': docsData,
+        'aposAttachments.json': attachmentsData
+        // attachments: 'attachments/' // TODO: add attachment into an "/attachments" folder


ETLaurent · 2023-08-23T16:26:03Z

lib/methods/export.js

+
+      const data = {
+        'aposDocs.json': docsData,
+        'aposAttachments.json': attachmentsData


aposAttachment.json has mocked data, for now (see TODOs)

ETLaurent · 2023-08-23T16:26:53Z

lib/methods/export.js

+      setTimeout(() => {
+        self.apos.attachment.uploadfs.remove(downloadPath, error => {
+          if (error) {
+            self.apos.util.error(error);
+          }
+        });
+      }, expiration || 1000 * 60 * 60);


setTimeout is bad... if the process gets killed, that won't be clearing the downloads

So we should find a better way to clean files.. is this cleaning files every hours?
Couldn't we clean the file when the client downloaded it by performing a new request with the file ID? 🤔

Every hour yes.

The client is requesting the file directly, from the public folder, and I don't think apostrophe handles that.
So I fear we cannot know if the file has been downloaded.

I may be wrong though...
It might be possible, I'm quickly looking for a possible solution, but I don't think we can.

Besides, we might want the archive to reside in public to be available for some time in the case the first download fails, IDK...

And if we're implementing a solution that's piping the whole thing (without writing the archvive on the server but directly streaming it to the client), we won't have this problem

ETLaurent · 2023-08-23T16:27:46Z

lib/methods/export.js

+      return {
+        url: downloadUrl
+      };


return url, we're not forced to do this because the url is given and opened by the export-download event, but let's return something in case of...

ETLaurent · 2023-08-23T16:27:56Z

lib/methods/index.js

+// TODO: remove:
+const attachmentsMock = [ { foo: 'bar' } ];


ETLaurent · 2023-08-23T16:28:01Z

lib/methods/index.js

+      const relatedTypes = self.apos.launder.strings(req.body.relatedTypes);
+      const extension = self.apos.launder.string(req.body.extension, 'zip');
+
+      // TODO: add batchSize?


ETLaurent · 2023-08-23T16:28:10Z

lib/methods/index.js

+      if (!relatedTypes.length) {
+        const docs = await self.fetchActualDocs(req, allIds, reporting);
+
+        // TODO: change undefined to attachments


and this TODO

ETLaurent · 2023-08-23T16:28:18Z

lib/methods/index.js

+
+      const docs = await self.fetchActualDocs(req, [ ...allIds, ...allRelatedIds ], reporting);
+
+      // TODO: change undefined to attachments


and that TODO

ETLaurent · 2023-08-23T16:28:49Z

lib/methods/index.js

+        .map(relatedDoc => relatedDoc._id);
+    },
+
+    // TODO: factorize with the one from AposI18nLocalize.vue


@ValJed this TODO can be removed when we'll merge our work

ETLaurent · 2023-08-23T16:28:54Z

lib/methods/index.js

+    },
+
+    // TODO: factorize with the one from AposI18nLocalize.vue
+    // TODO: limit recursion to 10 as we do when retrieving related types?


ETLaurent · 2023-08-23T16:29:06Z

lib/methods/index.js

+      return related.filter(doc => self.apos.modules[doc.type].relatedDocument !== false);
+    },
+
+    // TODO: use actual attachments when calling this method:


and that TODO

ETLaurent · 2023-08-23T16:29:22Z

modules/@apostrophecms/import-export-piece-type/index.js

+          // FIXME: the progress notification is not always dismissed.
+          // Probably a fix that needs to be done in job core module.


ETLaurent · 2023-08-23T16:29:41Z

ui/apos/apps/index.js

+    window.apos.bus && window.apos.bus.$on('export-download', event => {
+      if (!event.url) {
+        return;
+      }
+      window.open(event.url, '_blank');
+    });


the event I was referring to above

ETLaurent · 2023-08-23T16:30:26Z

lib/apiRoutes.js

+        // TODO: do not include types that have the `relatedDocument = false`
+        // option in their module configuration


we should do that, let's keep the TODO

Should be done in my PR

ETLaurent · 2023-08-23T16:31:28Z

lib/methods/index.js

+      // FIXME: seems like reporting is not working very well,
+      // when calling failure on purpose, we have the progress bar
+      // at 200%... 🤷‍♂️


to verify when making final tests

ETLaurent · 2023-08-23T16:31:50Z

index.js

+    self.exportFormats = {
+      zip,
+      gzip,
+      ...(self.options.exportFormats || {})


projects can add custom formats

ETLaurent · 2023-08-23T16:32:49Z

modules/@apostrophecms/import-export-piece-type/index.js

@@ -36,7 +36,10 @@ module.exports = {
        export: {
          label: 'aposImportExport:export',
          messages: {
-            progress: 'aposImportExport:exporting'
+            progress: 'aposImportExport:exporting',
+            completed: 'aposImportExport:exported',


the "completed" message makes the job handle the display of the final notification (eg. "Exported 2 Articles"

ETLaurent · 2023-08-23T16:33:55Z

modules/@apostrophecms/import-export-piece-type/index.js

-            progress: 'aposImportExport:exporting'
+            progress: 'aposImportExport:exporting',
+            completed: 'aposImportExport:exported',
+            icon: 'database-export-icon',


already by default in the core, which is weird and too close to the piece-type-exporter module.

still set it here because it's this module responsibility to set that database export icon

ETLaurent · 2023-08-23T16:35:26Z

modules/@apostrophecms/import-export-piece-type/index.js

+            progress: 'aposImportExport:exporting',
+            completed: 'aposImportExport:exported',
+            icon: 'database-export-icon',
+            resultsEventName: 'export-download'


that is necessary in order for the job to include the results into a 'export-download' which will trigger the url opening in the frontend (see ui/apos/apps/index.js).

the core has been changed so that resultsEventName is taken into account (see apostrophecms/apostrophe#4270)

We will need to document properly all these changes before to release.

ETLaurent · 2023-08-23T16:36:20Z

ui/apos/components/AposExportModal.vue

+              <AposSelect
+                :choices="extensions"
+                :selected="extension"
+                @change="onExtensionChange"


yeah, we said only ZIP...
but since gzip was added for the sake of testing another library, let's keep these 2 formats

ETLaurent · 2023-08-23T16:37:27Z

ui/apos/components/AposExportModal.vue

+      const result = await window.apos.http.post(`${action}/${this.action}`, {
+        busy: true,
+        body: {
+          _ids: docsId,
+          relatedTypes,
+          messages: this.messages,
+          extension: this.extension
+        }
+      });


this actual API call to export docs

ValJed · 2023-08-24T08:15:11Z

lib/formats/archiver.js

+      archive.append(content, { name: filename });
+    }
+
+    archive.finalize();


this triggers the event finish?

ETLaurent force-pushed the pro-4261-export-docs branch from 997ef19 to 48bb490 Compare August 17, 2023 09:06

ETLaurent changed the title ~~call export route~~ export docs and attachments in zip file Aug 17, 2023

ETLaurent force-pushed the pro-4261-export-docs branch from 48bb490 to 2e848a7 Compare August 17, 2023 11:31

ETLaurent added 13 commits August 17, 2023 13:35

call export route

b4d55ba

do not include related types when toggle is off

84e0b6e

count computed

240743f

wip

766e112

retrieve related ids

0889929

use map

404be10

fetch all related docs and from every mode

93ff732

extract code to a getRelatedDocsIds method

3666b0f

refactor code, maybe too much

313cc87

fetch also related docs from published docs

3727d06

add small but important comment

a6fa62b

add todos

d434984

weg3grhrthtrh4et

ef7a790

ETLaurent force-pushed the pro-4261-export-docs branch from 2e848a7 to ef7a790 Compare August 17, 2023 11:39

ETLaurent commented Aug 17, 2023

View reviewed changes

lib/methods.js Outdated Show resolved Hide resolved

ETLaurent changed the title ~~export docs and attachments in zip file~~ export docs and related docs Aug 17, 2023

ETLaurent added 6 commits August 21, 2023 09:23

export zip - wip

fb596ff

export zip using adm-zip lib - wip

fa5b169

open export url

a7aa784

remove gzip header

d6d1f56

reporting - wip

7bd5a92

reporting - wip (based on all fetched ids)

41413a1

ETLaurent force-pushed the pro-4261-export-docs branch from aec7a5d to 41413a1 Compare August 23, 2023 07:23

ETLaurent added 2 commits August 23, 2023 09:52

reporting - based on the given ids on body, fix doc type in messages

ca52fda

reporting - use core completed job notif

bc06522

ETLaurent added 3 commits August 23, 2023 17:04

add gzip format

37306ec

migrate zip lib to archiver

568ace9

add gzip to frontend selector

26c10d1

ETLaurent commented Aug 23, 2023

View reviewed changes

index.js

self.exportFormats = {

zip,

gzip,

...(self.options.exportFormats || {})

Copy link

Contributor Author

ETLaurent Aug 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

projects can add custom formats

ETLaurent commented Aug 23, 2023

View reviewed changes

ETLaurent requested a review from ValJed August 23, 2023 16:45

ETLaurent marked this pull request as ready for review August 23, 2023 16:45

re-organise and rename methods

a814186

ValJed approved these changes Aug 24, 2023

View reviewed changes

lib/formats/archiver.js

archive.append(content, { name: filename });

}

archive.finalize();

Copy link

Contributor

ValJed Aug 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this triggers the event finish?

ETLaurent merged commit 552a832 into main Aug 24, 2023
9 checks passed

ETLaurent deleted the pro-4261-export-docs branch August 24, 2023 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

export docs and related docs #7

export docs and related docs #7

ETLaurent commented Aug 15, 2023 •

edited

Loading

linear bot commented Aug 15, 2023

Generate files

Generate Zip

Progress bar notification

Permissions

ETLaurent Aug 17, 2023 •

edited

Loading

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023 •

edited

Loading

ETLaurent Aug 23, 2023

ValJed Aug 24, 2023 •

edited

Loading

ETLaurent Aug 24, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ValJed Aug 24, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ValJed Aug 24, 2023

ETLaurent Aug 23, 2023

ETLaurent Aug 23, 2023

ValJed Aug 24, 2023


		const docs = await self.fetchActualDocs(req, [ ...allIds, ...allRelatedIds ], reporting);

		// TODO: change undefined to attachments

		// FIXME: the progress notification is not always dismissed.
		// Probably a fix that needs to be done in job core module.

		// TODO: do not include types that have the `relatedDocument = false`
		// option in their module configuration

export docs and related docs #7

export docs and related docs #7

Conversation

ETLaurent commented Aug 15, 2023 • edited Loading

Summary

What are the specific steps to test this change?

What kind of change does this PR introduce?

Make sure the PR fulfills these requirements:

linear bot commented Aug 15, 2023

Generate files

Generate Zip

Progress bar notification

Permissions

ETLaurent Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ETLaurent Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValJed Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ETLaurent commented Aug 15, 2023 •

edited

Loading

ETLaurent Aug 17, 2023 •

edited

Loading

ETLaurent Aug 23, 2023 •

edited

Loading

ValJed Aug 24, 2023 •

edited

Loading