New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export CSV does not work with boost module #7026

Closed
thaihuutam opened this Issue Aug 1, 2017 · 19 comments

Comments

Projects
None yet
9 participants
@thaihuutam

thaihuutam commented Aug 1, 2017

Expected behaviour

Exporting CSV successfully with huge data set.

Actual behaviour

Cannot export CSV with huge data set, which boost module included.

Live demo with steps to reproduce

Example: https://jsfiddle.net/ubytofud/
This chart includes boost module v5.0.14, where the seriesThreshold is set to 50. There are 50 series in the chart, then the boost module kick in. Try the function Download CSV file, the downloaded file is contains no data rather than the header.

Affected browser(s)

Firefox 52.2.0

@vutienthinh

This comment has been minimized.

vutienthinh commented Aug 1, 2017

I also have exact issue. Could anyone help me out?

@sebastianbochan

This comment has been minimized.

Contributor

sebastianbochan commented Aug 1, 2017

The export module extracts points from series objects, but in the boost this reference is empty so you should extrac from userOptions.

Workaround:

@KacperMadej KacperMadej added the Bug label Aug 1, 2017

@thaihuutam

This comment has been minimized.

thaihuutam commented Aug 4, 2017

Thank @sebastianbochan for the workaround.
I just wonder if HighCharts has plan to fix this bug, or we have to live with the workaround.

@KacperMadej

This comment has been minimized.

Contributor

KacperMadej commented Aug 4, 2017

@thaihuutam The issue is open and labeled bug, so we will be working on resolving the problem - we are planning to fix this bug. In case the workaround is not working for you please let us know - the more info we have the better the fix could be provided.

@TorsteinHonsi

This comment has been minimized.

Collaborator

TorsteinHonsi commented Aug 7, 2017

Here is as far as I got now - this version passes all tests except one. Note that the most updated version of the module is https://github.com/highcharts/highcharts/blob/master/js/modules/export-data.src.js .

Highcharts.Chart.prototype.getDataRows = function () {
    var options = (this.options.exporting || {}).csv || {},
        xAxis,
        xAxes = this.xAxis,
        rows = {},
        rowArr = [],
        dataRows,
        names = [],
        i,
        x,
        xTitle,
        // Options
        dateFormat = options.dateFormat || '%Y-%m-%d %H:%M:%S',
        columnHeaderFormatter = options.columnHeaderFormatter || function (item, key, keyLength) {
            if (item instanceof Highcharts.Axis) {
                return (item.options.title && item.options.title.text) ||
                    (item.isDatetimeAxis ? 'DateTime' : 'Category');
            }
            return item ?
                item.name + (keyLength > 1 ? ' (' + key + ')' : '') :
                'Category';
        },
        xAxisIndices = [];

    // Loop the series and index values
    i = 0;
    each(this.series, function (series) {
        var keys = series.options.keys,
            pointArrayMap = keys || series.pointArrayMap || ['y'],
            valueCount = pointArrayMap.length,
            requireSorting = series.requireSorting,
            categoryMap = {},
            xAxisIndex = Highcharts.inArray(series.xAxis, xAxes),
            j,
            // A series-like object that has its own X increment
            mockSeries = {
                options: series.options,
                autoIncrement: series.autoIncrement
            };

        // Map the categories for value axes
        each(pointArrayMap, function (prop) {
            categoryMap[prop] = (series[prop + 'Axis'] && series[prop + 'Axis'].categories) || [];
        });

        if (series.options.includeInCSVExport !== false && series.visible !== false) { // #55

            // Build a lookup for X axis index and the position of the first
            // series that belongs to that X axis. Includes -1 for non-axis
            // series types like pies.
            if (!Highcharts.find(xAxisIndices, function (index) {
                return index[0] === xAxisIndex;
            })) {
                xAxisIndices.push([xAxisIndex, i]);
            }

            // Add the column headers, usually the same as series names
            j = 0;
            while (j < valueCount) {
                names.push(columnHeaderFormatter(series, pointArrayMap[j], pointArrayMap.length));
                j++;
            }

            each(series.options.data, function (pointConfig, pIdx) {
                var point = series.pointClass.prototype
                        .applyOptions.call(
                            { series: mockSeries },
                            pointConfig
                        ),
                    key = requireSorting ? point.x : point.x + '|' + pIdx,
                    prop,
                    val;

                j = 0;

                if (!rows[key]) {
                    // Generate the row
                    rows[key] = [];
                    // Contain the X values from one or more X axes
                    rows[key].xValues = [];
                }
                rows[key].x = point.x;
                rows[key].xValues[xAxisIndex] = point.x;
                
                // Pies, funnels, geo maps etc. use point name in X row
                if (!series.xAxis || series.exportKey === 'name') {
                    rows[key].name = point.name;
                }

                while (j < valueCount) {
                    prop = pointArrayMap[j]; // y, z etc
                    val = point[prop];
					// Pick a Y axis category if present
                    rows[key][i + j] = pick(categoryMap[prop][val], val);
                    j++;
                }

            });
            i = i + j;
        }
    });

    // Make a sortable array
    for (x in rows) {
        if (rows.hasOwnProperty(x)) {
            rowArr.push(rows[x]);
        }
    }

    var xAxisIndex, column;
    dataRows = [names];

    i = xAxisIndices.length;
    while (i--) { // Start from end to splice in
        xAxisIndex = xAxisIndices[i][0];
        column = xAxisIndices[i][1];
        xAxis = xAxes[xAxisIndex];

        // Sort it by X values
        rowArr.sort(function (a, b) { // eslint-disable-line no-loop-func
            return a.xValues[xAxisIndex] - b.xValues[xAxisIndex];
        });

        // Add header row
        xTitle = columnHeaderFormatter(xAxis);
        //dataRows = [[xTitle].concat(names)];
        dataRows[0].splice(column, 0, xTitle);

        // Add the category column
        each(rowArr, function (row) { // eslint-disable-line no-loop-func
            var category = row.name;
            if (!category) {
                if (xAxis.isDatetimeAxis) {
                    if (row.x instanceof Date) {
                        row.x = row.x.getTime();
                    }
                    category = Highcharts.dateFormat(dateFormat, row.x);
                } else if (xAxis.categories) {
                    category = pick(
                        xAxis.names[row.x],
                        xAxis.categories[row.x],
                        row.x
                    );
                } else {
                    category = row.x;
                }
            }

            // Add the X/date/category
            row.splice(column, 0, category);
        });
    }
    dataRows = dataRows.concat(rowArr);

    return dataRows;
};
@thaihuutam

This comment has been minimized.

thaihuutam commented Aug 16, 2017

@KacperMadej FYI, the workaround works for our case. Thanks a lot.
Still I will wait for the coming version of HighCharts where it is fixed.

@o3o

This comment has been minimized.

o3o commented Oct 30, 2017

I also have this issue. I will wait for the new version that fix it.
Thank you

@sangvvan

This comment has been minimized.

sangvvan commented Jan 30, 2018

Hi @KacperMadej : Do you have new version to fix this problems. It still happens on my side
Thank you

@KacperMadej

This comment has been minimized.

Contributor

KacperMadej commented Jan 30, 2018

Hi @sangvvan

The solutions presented in the thread are not passing all tests, that's why are not implemented.

@sangvvan

This comment has been minimized.

sangvvan commented Mar 12, 2018

Hi @TorsteinHonsi
I saw this issue is closed by you.
Was this issue fixed?
Thank you

@pawelfus

This comment has been minimized.

Contributor

pawelfus commented Mar 12, 2018

Hi @sangvvan

Yes, the issue is fixed and will be included in the next release. You can see pre-release version using our github.highcharts.com: https://jsfiddle.net/BlackLabel/ubytofud/5/

@sangvvan

This comment has been minimized.

sangvvan commented Mar 14, 2018

Hi @pawelfus
Thanks for your anwser.
Can you tell me how to get how to get minified for release ? Because the link above which coding isnot minified.
Thanks

@TorsteinHonsi

This comment has been minimized.

Collaborator

TorsteinHonsi commented Mar 14, 2018

You can minify it yourself at https://closure-compiler.appspot.com/home.

Just be aware this is pre-release code.

@sangvvan

This comment has been minimized.

sangvvan commented Mar 14, 2018

Thanks so much

@LeifBloomquist

This comment has been minimized.

LeifBloomquist commented Nov 9, 2018

I am running into this exact same issue. I pointed my highcharts src to https://github.highcharts.com as per the jsfiddle above to try out the latest version, but now I get the following error with large data sets:

Uncaught Failed to convert to blob
Highcharts.downloadURL @ export-data.js:216
Highcharts.Chart.downloadCSV @ export-data.js:1037
onclick @ export-data.js:1175
onclick @ exporting.js:1479

@TorsteinHonsi

This comment has been minimized.

Collaborator

TorsteinHonsi commented Nov 12, 2018

This fix was released months ago, so you don't have to use github.highcharts.com. Here's the originally posted fiddle with newest official Highcharts version: https://jsfiddle.net/highcharts/ubytofud/10/.

@LeifBloomquist

This comment has been minimized.

LeifBloomquist commented Nov 12, 2018

Ah, OK thanks. However the fiddle you posted doesn't seem to use boost (no boost: enabled: true?).

Here is a fiddle that shows my issue. When I select Download CSV, I get Failed- Network Error in Chrome 70.0.3538.77

https://jsfiddle.net/LeifBloomquist/5nepwg4s/

@TorsteinHonsi

This comment has been minimized.

Collaborator

TorsteinHonsi commented Nov 13, 2018

It's probably the amount of data that does it. It also happens with your fiddle even if the boost is disabled.

We would expect the CSV to be identical regardless of boost or not. I binary-searched for the tipping point, and it turns out that both with and without boost, the download fails when there are 37079 points or more in that chart:

for (var i= 0; i< 37079;  i++) { 
@LeifBloomquist

This comment has been minimized.

LeifBloomquist commented Nov 13, 2018

OK thanks. So how can I then resolve this, or work around it? I was going to write my own CSV-exporter, but with boost enabled the data points are lost after rendering I believe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment