Feat/export xlsx #2658

LouisDelbosc · 2022-07-12T15:37:40Z

Change Summary

Add export xlsx button and on nc-gui.
Here the issue#2565

Change type

feat: (new feature for the user, not a new feature for build script)
fix: (bug fix for the user, not a fix to a build script)
docs: (changes to the documentation)
style: (formatting, missing semi colons, etc; no production code change)
refactor: (refactoring production code, eg. renaming a variable)
test: (adding missing tests, refactoring tests; no production code change)
chore: (updating grunt tasks etc; no production code change)

Test/ Verification

Add new routes to receives csv raw data (instead of files)
Add option on More button to have an xlsx export
Modify sdk for the client to handle the new export type

Additional information / screenshots (optional)

I decide to have the file construction done by frontend because it was simpler for the backend logic (not duplicate or refactor convoluted code which can be fragile).
If you would like another way of doing things, let me know and I'd gladly implement them.

o1lab · 2022-07-12T16:47:18Z

Thanks for the PR - appreciate it.

However, expectation of export to excel here can mean export the entire database.
Since we are already supporting csv export - importing that csv to excel is a straightforward process.
Please get in touch with us before taking a feature implementation in discord.

LouisDelbosc · 2022-07-12T17:24:01Z

However, expectation of export to excel here can mean export the entire database.
Since we are already supporting csv export - importing that csv to excel is a straightforward process.
Please get in touch with us before taking a feature implementation in discord.

Thanks for the answer. I have numerous users (and potential users) on my nocodb instance who don't know how to import a CSV on excel.
For you and me, it is a straightforward process, but since nocodb want to be a no code solution and since I have this issue with my users and the many more who will follow, I thought it would be a good idea to have it here.

I'm open for any critics and way to solves this problem. If you know a way to do it without having to contribute, I'd be very pleased.

wingkwong

Other comments.

Extra empty cells
Discrepancy between exported data (See LastUpdated)

wingkwong · 2022-07-13T04:21:34Z

packages/nocodb/src/lib/utils/projectAcl.ts

@@ -26,6 +26,7 @@ export default {
    dataGroupBy: true,
    commentsCount: true,
    exportCsv: true,
+    exportCsvData: true,


What's difference with between exportCsv and exportCsvData? Why not using exportExcel?

The xlsx file is created on the frontend. The backend send data as CSV (just data, not file) and the frontend use this data to build a XLSX file.
That's why the endpoint I created is not excel or xlsx because the backend doesn't know about all theses things, it only sends CSV file or CSV data.

wingkwong · 2022-07-13T04:24:57Z

packages/nocodb-sdk/src/lib/globals.ts

@@ -13,6 +13,7 @@ export enum RelationTypes {

 export enum ExportTypes {
  CSV = 'csv',
+  CSV_DATA = 'csvData',


Should it be EXCEL?

As I said, the backend doesn't build the excel file. But maybe it'd be better to do so ?

wingkwong · 2022-07-13T04:28:14Z

packages/nocodb-sdk/src/lib/Api.ts

@@ -2554,7 +2554,7 @@ export class Api<
      orgs: string,
      projectName: string,
      tableName: string,
-      type: 'csv' | 'excel',
+      type: 'csv' | 'excel' | 'csvData',


Why adding a new one instead of using existing excel?

Because I made the backend send data instead of file.

wingkwong · 2022-07-13T04:28:30Z

packages/nocodb-sdk/src/lib/Api.ts

@@ -2917,7 +2917,7 @@ export class Api<
      projectName: string,
      tableName: string,
      viewName: string,
-      type: 'csv' | 'excel',
+      type: 'csv' | 'excel' | 'xlsx',


When to use xlsx and when to use csvData?

wingkwong · 2022-07-13T04:33:50Z

packages/nocodb/src/lib/meta/api/exportApis.ts

@@ -18,6 +18,19 @@ async function exportCsv(req: Request, res: Response) {
  res.send(data);
 }

+async function exportCsvData(req: Request, res: Response) {


probably make a generic function, i.e. combining into exportCsv

wingkwong · 2022-07-13T04:48:45Z

packages/nocodb/src/lib/meta/api/dataApis/dataAliasExportApis.ts

@@ -39,4 +56,10 @@ router.get(
  ncMetaAclMw(csvDataExport, 'exportCsv')
 );

+router.get(
+  '/api/v1/db/data/:orgs/:projectName/:tableName/views/:viewName/export/csvData',


naming - similar to other comments.

LouisDelbosc · 2022-07-13T15:52:18Z

Other comments.

Extra empty cells

Discrepancy between exported data (See LastUpdated)

It seems to be the case for csv spreadsheet, it is an issue with xlsx ?
I did not see that beforehand thank you. After looking for information, it seems excel store datetime using numbers, I check on numbers or google spreadsheet and it doesn't seems to works as wanted. I need a real excel to check if we have a datetime reading this information. Otherwise, i'm quite disappointed :(

yohanboniface · 2022-07-18T13:38:08Z

I did not see that beforehand thank you. After looking for information, it seems excel store datetime using numbers, I check on numbers or google spreadsheet and it doesn't seems to works as wanted. I need a real excel to check if we have a datetime reading this information. Otherwise, i'm quite disappointed :(

Are you setting the right cell type ?

mertmit

Thank you for the PR, left some comments waiting for your update. You can ask for a review when you are done. Feel free to ask anything on process.

mertmit · 2022-07-18T14:33:36Z

packages/nc-gui/components/project/spreadsheet/components/MoreActions.vue

+            res = await this.$api.public.csvExport(this.publicViewId, ExportTypes.CSV_DATA, {
+              query: {
+                fields:
+                  this.queryParams &&
+                  this.queryParams.fieldsOrder &&
+                  this.queryParams.fieldsOrder.filter(c => this.queryParams.showFields[c]),
+                offset,
+                sortArrJson: JSON.stringify(
+                  this.reqPayload &&
+                    this.reqPayload.sorts &&
+                    this.reqPayload.sorts.map(({ fk_column_id, direction }) => ({
+                      direction,
+                      fk_column_id,
+                    }))
+                ),
+                filterArrJson: JSON.stringify(this.reqPayload && this.reqPayload.filters),
+              },
+              headers: {
+                'xc-password': this.reqPayload && this.reqPayload.password,
+              },
+            });


You should handle shared view case as well. Nothing exported + no corresponding endpoint on backend.

mertmit · 2022-07-18T14:35:07Z

packages/nc-gui/components/project/spreadsheet/components/MoreActions.vue

+            const { data } = res
+            const xlsx = XLSX.read(data.csvData, { type: 'string' })
+            offset = +res.headers['nc-export-offset']
+            XLSX.writeFile(xlsx, `${this.meta.title}_exported_${c++}.xlsx`)
+            if (offset > -1) {
+              this.$toast.info('Downloading more files').goAway(3000)
+            } else {
+              this.$toast.success('Successfully exported all table data').goAway(3000)
+            }
+          }


This part should be out of if/else block, we should return file for both cases.

mertmit · 2022-07-18T14:39:15Z

packages/nc-gui/components/project/spreadsheet/components/MoreActions.vue

+              ExportTypes.CSV_DATA
+            )
+            const { data } = res
+            const xlsx = XLSX.read(data.csvData, { type: 'string' })


Seems like you can use raw: true to avoid unwanted processing of dates, also looks like it fixes extra empty cells.

LouisDelbosc added 4 commits July 12, 2022 17:20

feat: add new endpoint to get raw csv data instead of file

b8e77f1

feat: add new button to get raw csv data and transform it to xlsx file

b3fba90

feat: add translations for xlsx download button

2fe3d06

feat: update sdk client to handle new endpoints

1d306a3

o1lab requested a review from wingkwong July 12, 2022 18:29

wingkwong reviewed Jul 13, 2022

View reviewed changes

wingkwong requested a review from mertmit July 13, 2022 07:47

mertmit requested changes Jul 18, 2022

View reviewed changes

LouisDelbosc closed this Jul 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/export xlsx #2658

Feat/export xlsx #2658

LouisDelbosc commented Jul 12, 2022

o1lab commented Jul 12, 2022

LouisDelbosc commented Jul 12, 2022

wingkwong left a comment

wingkwong Jul 13, 2022

LouisDelbosc Jul 13, 2022

wingkwong Jul 13, 2022

LouisDelbosc Jul 13, 2022

wingkwong Jul 13, 2022

LouisDelbosc Jul 13, 2022

wingkwong Jul 13, 2022

wingkwong Jul 13, 2022

wingkwong Jul 13, 2022

LouisDelbosc commented Jul 13, 2022

yohanboniface commented Jul 18, 2022

mertmit left a comment

mertmit Jul 18, 2022

mertmit Jul 18, 2022

mertmit Jul 18, 2022

Feat/export xlsx #2658

Feat/export xlsx #2658

Conversation

LouisDelbosc commented Jul 12, 2022

Change Summary

Change type

Test/ Verification

Additional information / screenshots (optional)

o1lab commented Jul 12, 2022

LouisDelbosc commented Jul 12, 2022

wingkwong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LouisDelbosc commented Jul 13, 2022

yohanboniface commented Jul 18, 2022

mertmit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment