Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lens] CSV Export for Lens #83430

Merged
merged 29 commits into from
Nov 24, 2020
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
078e554
:sparkles: First draft of csv export in Lens
dej611 Nov 16, 2020
5d789d6
:alembic: Create action stub
dej611 Nov 16, 2020
965bb69
:truck: + :white_check_mark: Move export csv to data plugins + basic …
dej611 Nov 17, 2020
456d14a
:lipstick: Enable/disable rather than show/hide button
dej611 Nov 17, 2020
8f917b1
:bug: Fix tests
dej611 Nov 17, 2020
851b98f
:speech_balloon: Fix i18n checks + :packages: Update API docs
dej611 Nov 17, 2020
08d75d0
:fire: removed embeddable action for now
dej611 Nov 17, 2020
12ac1e0
Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…
dej611 Nov 17, 2020
88eefc6
:bug: Forgotten type
dej611 Nov 18, 2020
ce68a93
Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…
dej611 Nov 18, 2020
88c2a8d
:lipstick: Move the cancel button on the right
dej611 Nov 18, 2020
27678f4
:bug: Avoid to save the activeData content
dej611 Nov 18, 2020
52f606f
:bug: Fix tests
dej611 Nov 18, 2020
237664e
Merge branch 'master' into feature/lens/csv-export
kibanamachine Nov 19, 2020
4de7ae8
:sparkles: split the current plugin in two distinct areas
dej611 Nov 19, 2020
6e88319
Merge branch 'feature/lens/csv-export' of github.com:dej611/kibana in…
dej611 Nov 19, 2020
f851e69
:memo: Update API documentation
dej611 Nov 20, 2020
3dad985
:ok_hand: Integrated feedback from review
dej611 Nov 20, 2020
5749f32
:label: fix type issue
dej611 Nov 20, 2020
4674a83
:white_check_mark: Add download unit tests
dej611 Nov 20, 2020
3da4fab
Merge branch 'master' into feature/lens/csv-export
kibanamachine Nov 23, 2020
4cada29
:ok_hand: Integrate feedback
dej611 Nov 23, 2020
1fbdeef
Merge branch 'feature/lens/csv-export' of github.com:dej611/kibana in…
dej611 Nov 23, 2020
084a22f
:memo: Update API doc
dej611 Nov 23, 2020
cb173b0
Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…
dej611 Nov 23, 2020
ca868b3
:recycle: Refactor export plugin as per feedback received
dej611 Nov 23, 2020
0e7770c
:memo: Update API doc
dej611 Nov 23, 2020
80eac9f
:ok_hand: use named exports
dej611 Nov 23, 2020
c3941a1
:white_check_mark: Add basic functional test
dej611 Nov 24, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-plugins-data-public](./kibana-plugin-plugins-data-public.md) &gt; [CSV\_MIME\_TYPE](./kibana-plugin-plugins-data-public.csv_mime_type.md)

## CSV\_MIME\_TYPE variable

<b>Signature:</b>

```typescript
CSV_MIME_TYPE = "text/plain;charset=utf-8"
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-plugins-data-public](./kibana-plugin-plugins-data-public.md) &gt; [exportAsCSVs](./kibana-plugin-plugins-data-public.exportascsvs.md)

## exportAsCSVs() function

<b>Signature:</b>

```typescript
export declare function exportAsCSVs(filename: string, datatables: Record<string, Datatable> | undefined, options: CSVOptions): Record<string, {
content: string;
type: string;
}> | undefined;
```

## Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| filename | <code>string</code> | filename to use (either as is, or as prefix for multiple CSVs) for the files to download |
| datatables | <code>Record&lt;string, Datatable&gt; &#124; undefined</code> | data (as a dictionary of Datatable) to be translated into CSVs. It can contain multiple tables. |
| options | <code>CSVOptions</code> | set of options for the exporter |

<b>Returns:</b>

`Record<string, {
content: string;
type: string;
}> | undefined`

A dictionary of files to download: the key is the filename and the value the CSV string

Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@

| Function | Description |
| --- | --- |
| [exportAsCSVs(filename, datatables, options)](./kibana-plugin-plugins-data-public.exportascsvs.md) | |
| [getDefaultQuery(language)](./kibana-plugin-plugins-data-public.getdefaultquery.md) | |
| [getEsPreference(uiSettings, sessionId)](./kibana-plugin-plugins-data-public.getespreference.md) | |
| [getSearchParamsFromRequest(searchRequest, dependencies)](./kibana-plugin-plugins-data-public.getsearchparamsfromrequest.md) | |
Expand Down Expand Up @@ -104,6 +105,7 @@
| [castEsToKbnFieldTypeName](./kibana-plugin-plugins-data-public.castestokbnfieldtypename.md) | Get the KbnFieldType name for an esType string |
| [connectToQueryState](./kibana-plugin-plugins-data-public.connecttoquerystate.md) | Helper to setup two-way syncing of global data and a state container |
| [createSavedQueryService](./kibana-plugin-plugins-data-public.createsavedqueryservice.md) | |
| [CSV\_MIME\_TYPE](./kibana-plugin-plugins-data-public.csv_mime_type.md) | |
| [ES\_SEARCH\_STRATEGY](./kibana-plugin-plugins-data-public.es_search_strategy.md) | |
| [esFilters](./kibana-plugin-plugins-data-public.esfilters.md) | |
| [esKuery](./kibana-plugin-plugins-data-public.eskuery.md) | |
Expand Down
117 changes: 117 additions & 0 deletions src/plugins/data/public/exports/export_csv.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import { Datatable } from 'src/plugins/expressions';
import { FieldFormat } from '../../common/field_formats';
import { CSV_MIME_TYPE, exportAsCSVs } from './export_csv';

function getDefaultOptions() {
const formatFactory = jest.fn();
formatFactory.mockReturnValue({ convert: (v: unknown) => `Formatted_${v}` } as FieldFormat);
return {
csvSeparator: ',',
quoteValues: true,
formatFactory,
};
}

function getDataTable({
multipleLayers,
multipleColumns,
}: { multipleLayers?: boolean; multipleColumns?: boolean } = {}): Record<string, Datatable> {
const datatables: Record<string, Datatable> = {
layer1: {
type: 'datatable',
columns: [{ id: 'col1', name: 'columnOne', meta: { type: 'string' } }],
rows: [{ col1: 'value' }],
},
};
if (multipleColumns) {
datatables.layer1.columns.push({ id: 'col2', name: 'columnTwo', meta: { type: 'number' } });
datatables.layer1.rows[0].col2 = 5;
}
if (multipleLayers) {
datatables.layer2 = {
type: 'datatable',
columns: [{ id: 'col1', name: 'columnOne', meta: { type: 'string' } }],
rows: [{ col1: 'value' }],
};
}
return datatables;
}

describe('CSV exporter', () => {
test('should do nothing with no data', () => {
expect(exportAsCSVs('noData', undefined, getDefaultOptions())).toStrictEqual(undefined);
});

test('should not break with empty data', () => {
expect(exportAsCSVs('emptyFile', {}, getDefaultOptions())).toStrictEqual({});
});

test('should export formatted values by default', () => {
expect(exportAsCSVs('oneCSV', getDataTable(), getDefaultOptions())).toStrictEqual({
'oneCSV.csv': { content: 'columnOne\r\n"Formatted_value"\r\n', type: CSV_MIME_TYPE },
});
});

test('should not quote values when requested', () => {
return expect(
exportAsCSVs('oneCSV', getDataTable(), { ...getDefaultOptions(), quoteValues: false })
).toStrictEqual({
'oneCSV.csv': { content: 'columnOne\r\nFormatted_value\r\n', type: CSV_MIME_TYPE },
});
});

test('should use raw values when requested', () => {
expect(
exportAsCSVs('oneCSV', getDataTable(), { ...getDefaultOptions(), raw: true })
).toStrictEqual({
'oneCSV.csv': { content: 'columnOne\r\nvalue\r\n', type: CSV_MIME_TYPE },
});
});

test('should use separator for multiple columns', () => {
expect(
exportAsCSVs('oneCSV', getDataTable({ multipleColumns: true }), getDefaultOptions())
).toStrictEqual({
'oneCSV.csv': {
content: 'columnOne,columnTwo\r\n"Formatted_value","Formatted_5"\r\n',
type: CSV_MIME_TYPE,
},
});
});

test('should support multiple layers', () => {
expect(
exportAsCSVs('twoCSVs', getDataTable({ multipleLayers: true }), getDefaultOptions())
).toStrictEqual({
'twoCSVs-1.csv': { content: 'columnOne\r\n"Formatted_value"\r\n', type: CSV_MIME_TYPE },
'twoCSVs-2.csv': { content: 'columnOne\r\n"Formatted_value"\r\n', type: CSV_MIME_TYPE },
});
});

test('should escape values', () => {
const datatables = getDataTable();
datatables.layer1.rows[0].col1 = '"value"';
expect(exportAsCSVs('oneCSV', datatables, getDefaultOptions())).toStrictEqual({
'oneCSV.csv': { content: 'columnOne\r\n"Formatted_""value"""\r\n', type: CSV_MIME_TYPE },
});
});
});
119 changes: 119 additions & 0 deletions src/plugins/data/public/exports/export_csv.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

// Inspired by the inspector CSV exporter

import { FormatFactory } from 'src/plugins/data/common/field_formats/utils';
import { Datatable } from 'src/plugins/expressions';
import { DownloadableContent } from 'src/plugins/share/public/';

const LINE_FEED_CHARACTER = '\r\n';
const nonAlphaNumRE = /[^a-zA-Z0-9]/;
const allDoubleQuoteRE = /"/g;
export const CSV_MIME_TYPE = 'text/plain;charset=utf-8';

// TODO: enhance this later on
function escape(val: object | string, quoteValues: boolean) {
if (val != null && typeof val === 'object') {
val = val.valueOf();
}

val = String(val);

if (quoteValues && nonAlphaNumRE.test(val)) {
val = `"${val.replace(allDoubleQuoteRE, '""')}"`;
}

return val;
}

interface CSVOptions {
csvSeparator: string;
quoteValues: boolean;
formatFactory: FormatFactory;
raw?: boolean;
}

function buildCSV(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets rename this one to datatableToCSV and export it

{ columns, rows }: Datatable,
{ csvSeparator, quoteValues, formatFactory, raw }: Omit<CSVOptions, 'asString'>
) {
// Build the header row by its names
const header = columns.map((col) => escape(col.name, quoteValues));

const formatters = columns.reduce<Record<string, ReturnType<FormatFactory>>>(
(memo, { id, meta }) => {
memo[id] = formatFactory(meta?.params);
return memo;
},
{}
);

// Convert the array of row objects to an array of row arrays
const csvRows = rows.map((row) => {
return columns.map((column) =>
escape(raw ? row[column.id] : formatters[column.id].convert(row[column.id]), quoteValues)
);
});

return (
[header, ...csvRows].map((row) => row.join(csvSeparator)).join(LINE_FEED_CHARACTER) +
LINE_FEED_CHARACTER
); // Add \r\n after last line
}

/**
*
* @param filename - filename to use (either as is, or as prefix for multiple CSVs) for the files to download
* @param datatables - data (as a dictionary of Datatable) to be translated into CSVs. It can contain multiple tables.
* @param options - set of options for the exporter
*
* @returns A dictionary of files to download: the key is the filename and the value the CSV string
*/
export function exportAsCSVs(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets change this function so it doesn't download the file but just returns csv as a string, this way we can move the function to common and make it usable from server.

the download aspect of it should be moved to some other place (possibly share plugin) so that can be used for other things as well, as that download function would now just take filename and filecontent in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You proposing splitting this into two exportCSVs + download plugins?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, two functions as converting to csv can be common and downloading can be used for other things beside csv as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am wondering if this function should be kept inside lens for now. how often will we want to convert multiple datatables at once ? also i am wondering if the reference to filename here should be omited but rather just use the datatable name ? @lukeelmers whats your opinion ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind that the DataTable type has no name property. The key:string passed for the record may be just internal ids (for instance in Lens it represents the layer id).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we move down the path of creating these various utility functions for datatables, I think it's important to have some consistency in how you interact with them. For any datatable utility function, I'd expect an interface similar to the following:

type SomeFn = (table: Datatable, options?: Record<string, whatever>) => Promise<Something> | Something;

how often will we want to convert multiple datatables at once ?

For the sake of consistency, I would vote to make this function operate on a single table just like our other datatable helpers.

i am wondering if the reference to filename here should be omited but rather just use the datatable name

I think the concept of a filename is only here because the file download functionality was extracted from this function originally. IMHO it would make more sense to remove it and handle it when calling the download helper from the share plugin.

This would make the interface something like:

exportAsCsv(table: Datatable, options: CSVOptions) => string;

Then Lens would handle calling it multiple times (for multiple tables), assigning a filename, and passing the output to the download helper.

To me this feels like the best way to make these utilities as decoupled as possible, and keeps the lens-specific stuff in lens. (And of course we can always revisit down the road if some of the lens logic is needed for a bunch of other apps)

filename: string,
datatables: Record<string, Datatable> | undefined,
options: CSVOptions
) {
if (datatables == null) {
return;
}
// build a csv for datatable layer
const csvs = Object.keys(datatables)
.filter((layerId) => {
return (
datatables[layerId].columns.length &&
datatables[layerId].rows.length &&
datatables[layerId].rows.every((row) => Object.keys(row).length)
);
})
.reduce<Record<string, string>>((memo, layerId) => {
memo[layerId] = buildCSV(datatables[layerId], options);
return memo;
}, {});

const layerIds = Object.keys(csvs);

return layerIds.reduce<Record<string, Exclude<DownloadableContent, Blob>>>((memo, layerId, i) => {
const content = csvs[layerId];
const postFix = layerIds.length > 1 ? `-${i + 1}` : '';
memo[`${filename}${postFix}.csv`] = { content, type: CSV_MIME_TYPE };
return memo;
}, {});
}
20 changes: 20 additions & 0 deletions src/plugins/data/public/exports/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

export * from './export_csv';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we make my proposed changes to exportAsCsv, then we could move this all to the common directory. (Maybe common/utils?) and export it from both public/index.ts and server/index.ts

6 changes: 6 additions & 0 deletions src/plugins/data/public/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,12 @@ export {
FieldFormat,
} from '../common';

/**
* Exporters (CSV)
*/

export * from './exports';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment re: using named exports from top-level public/index.ts and server/index.ts


/*
* Index patterns:
*/
Expand Down
Loading