Using puppeteer branch as a node module

[drmrbrewer]

I posted here and was advised that I should be using the puppeteer branch straight away, even though it's not released yet.

But @PaulDalek @cvasseng I'm seriously confused about how (or even whether it's possible) to use the new puppeteer branch as a node module. The README in this branch still seems to be referring to the phantomjs version for the node module part? Despite my best efforts, I cannot get it to work.

I think (but am not sure) that the only way to load the puppeteer branch via the package.json is to refer to a specific commit (or is it?), like so:

"dependencies": {
  "express": "^4.18.2",
  "highcharts-export-server": "git+https://github.com/highcharts/node-export-server.git#92e6f01989265a10079e2ad99e36d2d9208be2b0"
},

Then in my node app I'm using the following function as a GET route for an express app. I am deliberately showing use of a MyChart function/class for providing functionality relating to the chart generation (ideally providing the options and also the callback) because that is the model I want to use in practice. Please check in the code where I've added a "NOTE:" comment about how I'd really like to do things, rather than having to put customCode and callback in their own standalone files.

const highchartsPuppeteer = function (req, res, next) {

    function MyChart(jsondata, options) {
        this.jsondata = jsondata;
        this.options = options;
        this.enableLabel = true;
    }

    MyChart.prototype.getChartOptions = function () {
        const type = this.options.useSpline ? 'spline' : 'line';
        const data = this.jsondata;
        return {
            xAxis: {
                categories: ["Jan", "Feb", "Mar", "Apr", "Mar", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
            },
            series: [{
                type: type,
                data: data.seriesA
            }, {
                type: type,
                data: data.seriesB
            }]
        }
    };

    MyChart.prototype.addLabel = function (chart) {
        chart.renderer.label(
            'This label is added in the callback ' + Highcharts.version + " " + _.round(4.006, 2),
            100,
            100
        )
        .attr({
            fill : '#90ed7d',
            padding: 10,
            r: 10,
            zIndex: 10
        })
        .css({
            color: 'black',
            width: '100px'
        })
        .add();    
    };

    // NOTE: I really want to be using this as the callback function...
    MyChart.prototype.onChartLoad = function (chart) {
        if (this.enableLabel && this.options.showLabel) {
            this.addLabel(chart);
        }
    };

    const myChart = new MyChart({
        seriesA: [1, 3, 2, 4, 3],
        seriesB: [5, 3, 4, 2, 5]
    }, {
        useSpline: true,
        showLabel: true
    });

    const fileName = 'outfile.png';

    const exportSettings = {
        export: {
            type: 'png',
            outfile: fileName,
            options: myChart.getChartOptions()
        },
        customCode: {
            allowCodeExecution: true,
            allowFileResources: true,
            resources: './lib/lodash.min.js',
            // NOTE: really I want to be defining the customCode function directly, without reference to a file:
            /*
                customCode: function (options) {
                    options.title = { 
                        text: 'Changed through custom code ' + Highcharts.version + ' ' + _.round(4.006, 2)
                    };
                },
           */
           // but instead (for now) I'm trying:
           customCode: './lib/customCode.js',
           // NOTE: really I want to be defining the callback with reference to myChart above:
           /*
                callback: function (chart) {
                    myChart.onChartLoad(chart);
                }
           */   
           // but instead (for now) I'm trying:
           callback: './lib/callback.js'
        }
    };

    exporter.initPool();

    exporter.startExport(exportSettings, (info, error) => {
        if (error) {
            logger.error('error generating chart:', error);
            res.sendStatus(400);
            return;
        }
        // NOTE: really I'd prefer not to have any file generated... just send directly from the output of puppeteer...
        res.sendFile(fileName);
    });
};

File ./lib/lodash.min.js is just a copy/paste from here.

File ./lib/customCode.js:

function (options) {
    options.title = { 
        text: 'Changed through custom code ' + Highcharts.version + ' ' + _.round(4.006, 2)
    };
}

File ./lib/callback.js (though really I want the callback to come from MyChart.onChartLoad()):

function (chart) {
    chart.renderer.label(
        'This label is added in the callback ' + Highcharts.version + " " + _.round(4.006, 2),
        100,
        100
    )
    .attr({
        fill : '#90ed7d',
        padding: 10,
        r: 10,
        zIndex: 10
    })
    .css({
        color: 'black',
        width: '100px'
    })
    .add();
}

But running the app and calling the route, I'm just getting:

[verbose] - [chart] Starting exporting process.
[verbose] - [chart] Attempting to export from a raw input.
[notice] - [cli] No resources found.
[ERROR] error generating chart: {
  error: true,
  message: 'The callback, resources and customCode have been disabled for this server.'
}
[notice] - [cache] Fetching and caching Highcharts dependencies.
/usr/src/app/node_modules/highcharts-export-server/lib/cache.js:113
	...coreScripts.map((c) => `${hcVersion}${c}`),
				   ^
TypeError: Cannot read properties of undefined (reading 'map')
	at updateCache (/usr/src/app/node_modules/highcharts-export-server/lib/cache.js:113:20)
	at checkCache (/usr/src/app/node_modules/highcharts-export-server/lib/cache.js:236:28)
	at Object.initPool (/usr/src/app/node_modules/highcharts-export-server/lib/index.js:55:11)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Any hints about how to get this to work as a node module?

[drmrbrewer]

I've made some progress... the following are the changed parts (from const exportSettings onwards):

const exportSettings = {
    version: 'latest',
    cdnURL: 'https://code.highcharts.com/',
    highcharts: {
        coreScripts: [
            'highcharts',
            'highcharts-more'
        ],
        modules: [],
        indicators: [],
        scripts: [
            'https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.20/lodash.min.js'
        ]
    },
    export: {
        type: 'png',
        options: myChart.getChartOptions()
    },
    customCode: {
        allowCodeExecution: true,
        allowFileResources: true,
        resources: '../../lib/lodash.min.js',
        customCode: '../../lib/custom_code.js',
        callback: '../../lib/call_back.js'
    }
};

await exporter.initPool(exportSettings);

exporter.startExport(exportSettings, (info, error) => {
    if (error) {
        logger.error('error generating chart:', error);
        res.sendStatus(400);
        return;
    }

    const img = Buffer.from(info.data, 'base64');
    res.header('Content-Type', 'image/png');
    res.header('Content-Length', img.length);

    res.status(200).send(img);
});

Observations:

• you have to pass exportSettings to exporter.initPool() as well as exporter.startExport(), which I wasn't before
• you seem to have to await the call to exporter.initPool(), which I wasn't before
• you seem to have to define the elements of highcharts (i.e. coreScripts etc) explicitly and can't just rely on defaults being used if not... this is not ideal... shouldn't it just use sensible defaults if nothing is specified?
• you have to define the path for customCode and callback relative to the relevant folder within node_modules... this is not ideal because these files will in practice always be outside the installed node_modules won't they (because they are custom functions not library functions)... so we need to assume where we are within node_modules and then go up a few levels to get outside node_modules... but this might break in future if the structure within node_modules changes
• callback seems to work but I can't get customCode to work... in the examples below, the label is added but not the sun graphic
• I can't get my lodash dependency loaded so that its available within customCode and callback functions... the following example callback function refers to lodash function _.round() but the callback function cannot find it... how do I properly load lodash? I tried two different ways above (via scripts and resources). The chart renders fine when the call to _.round() is removed.
• I figured out that info.data in the function passed to exporter.startExport() contains the image data so I can send that directly without ever writing to a file

Here are the files I'm referring to above:

File ../../lib/call_back.js:

function callback(chart) {
    chart.renderer
        .label(
            'This label is added in the callback ' + Highcharts.version + ' ' + _.round(4.006, 2),
            100,
            100
        )
        .attr({
            id: 'renderer-callback-label',
            fill: '#90ed7d',
            padding: 10,
            r: 10,
            zIndex: 10
        })
        .css({
            color: 'black',
            width: '100px'
        })
        .add();
}

File ../../lib/custom_code.js:

Highcharts.setOptions({
    chart: {
        events: {
            render: function () {
                this.renderer
                    .image(
                        'https://www.highcharts.com/samples/graphics/sun.png',
                        100,
                        75,
                        20,
                        20
                    )
                    .add();
            }
        }
    }
});

Aside from the above observations, the main issue is that you cannot seem to define a callback or customCode function directly, but can only do so with reference to a separate file (with its own independent context). In other words, I can't define my callback like so with reference to my myChart object:

callback: function (chart) {
    myChart.onChartLoad(chart);
}

This is a major downside IMHO... kinda defeats the purpose of using this as a node module (with all the benefits and convenience that brings). Is there any way that this could be done?

Not only is it not convenient to force the callback into a separate file, but it also seems to be the case that a simple comment line will break everything, e.g. as follows, making it impractical for development purposes:

function callback(chart) {
    chart.renderer
        .label(
            // -------- this comment line will break things! --------
            'This label is added in the callback ' + Highcharts.version,
            100,
            100
        )
        .attr({
            id: 'renderer-callback-label',
            fill: '#90ed7d',
            padding: 10,
            r: 10,
            zIndex: 10
        })
        .css({
            color: 'black',
            width: '100px'
        })
        .add();
}

@PaulDalek @cvasseng

[PaulDalek]

Hi @drmrbrewer

Thank you for providing us with your insights and I'm sorry for the late reply. First of all, sorry for the mess with information on how to use exactly the new export server version. There were some changes that still aren't correctly described in the README section. We will improve this one.

I've looked at your code and analyzed it and here are the answers to your questions:

You have to pass exportSettings to exporter.initPool() as well as exporter.startExport(), which I wasn't before.

The first major change is that the initPool function now requires an object with options to be passed as an argument. The reason behind this is that we wanted to give a possibility for users to pass personalized options (e.g. all pool related configuration). For a new options structure, please take a look at the ./lib/schemas/config.js schema file. I must admit though that I didn't think through it when it comes to usage as a module. For now I've added the getDefaultOptions into the main module in order to get the defaults and merge them with the options provided by the user (I've added the latest commit at the bottom of the message). This should work fine until we figure out how to split the logic more reasonably. The reason behind errors that you've encountered was the lack of the default options and configuration, therefore the required Highcharts and other scripts couldn't be fetched.

You seem to have to await the call to exporter.initPool(), which I wasn't before.

Yes, some additional async code was added along the way.

You seem to have to define the elements of highcharts (i.e. coreScripts etc) explicitly and can't just rely on defaults being used if not...

We wanted to add more flexibility when it comes for users to specify any Highcharts code, which includes coreScripts ('highcharts', 'highcharts-more', 'highcharts-3d' core scripts), modules (all modules loaded in a right order), indicators (all indicators related code) and finally scripts (any additional scripts, e.g. moment.js or lodash in this case). Please, take a look at the ./lib/schemas/config.js file to see the default setting. When no scripts are found, the ones from arrays of mentioned objects will be fetched. The .cache folder will be created then which will hold information about currently used code along with downloaded sources (the sources.js file).

You have to define the path for customCode and callback relative to the relevant folder within node_modules.

Yes, you are right. That's an issue. For now I simply changed it to use absolute path but ofc. will correct it to make it work better.

Callback seems to work but I can't get customCode to work... in the examples below, the label is added but not the sun graphic.

It seems that the sun icon is blocked by the not same origin error even in the official demo (ERR_BLOCKED_BY_RESPONSE.NotSameOrigin). When you change the image for the circle for example, you will see that the renderer works correctly in the customCode section.

I can't get my lodash dependency loaded so that its available within customCode and callback functions.

To include the lodash script, you should place it in the scripts.value array, in the ./lib/schemas/config.js file. I know that it is a little bit unintuitive when it comes to using it as a node module but it was designed especially for the setting server usage.

I figured out that info.data in the function passed to exporter.startExport() contains the image data so I can send that directly without ever writing to a file

Sure, you will find the base64 representation of a chart in the info.data.

Also, here I explained the resources, callback and customCode properties in more detail. Let's take a look at the properties from the customCode section now:

• resources: Loading lodash won't work because of a few reasons. First, the resources should be an object of the following structure: { "js": "raw JS", "css": "raw CSS", "files": [array of js files] }. Please, take a look at the ./samples/resources/resources.json for reference. Also, the resources are loaded after setting the content of the page. I’ve mentioned how to load lodash in one of the above points.

• callback: Here, the usage didn't change much as in the old version you could load callback from a file or directly as a stringified version. This is dangerous so in order for this to work, you must set the allowCodeExecution flag to true to allow it (it should only be considered in a sandboxed environment).

• customCode: The customCode right now works as any JS code that is called before the chart creation. The same rules as in case of the callback applies here too.

The reason why function related options can only be passed that way is because later in the code they are inserted directly in the stringified template that is used when creating a page through Puppeteer headless instance.

As for your other observations:

The README in this branch still seems to be referring to the phantomjs version for the node module part?

I will take a look at this one and provide suggested corrections. It seems that the code snippet doesn't mention about the necessity of the default options for the initPool function along with some other minor inaccuracies.

I think (but am not sure) that the only way to load the puppeteer branch via the package.json is to refer to a specific commit (or is it?), like so

Yes, you are right. Unfortunately, the Puppeteer version still isn't available on the npm registry. For now you can use it as a downloaded repo or by referencing by commit (or with a local path). Here is the newest commit for you: 70e1d04.

Finally, here is a snippet of your code that I’ve prepared for you. It will correctly export a chart to the base64 representation, inside the info.data (named the package as puppeteer-highcharts-export-server for readability).

import puppeteerHighchartsExportServer from 'puppeteer-highcharts-export-server';

export default async function highchartsPuppeteer(req, res, next) {
  function MyChart(jsondata, options) {
    this.jsondata = jsondata;
    this.options = options;
    this.enableLabel = true;
  }

  MyChart.prototype.getChartOptions = function () {
    const type = this.options.useSpline ? 'spline' : 'line';
    const data = this.jsondata;

    return {
      xAxis: {
        categories: ["Jan", "Feb", "Mar", "Apr", "Mar", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
      },
      series: [{
        type: type,
        data: data.seriesA
      }, {
        type: type,
        data: data.seriesB
      }]
    };
  };

  MyChart.prototype.addLabel = function (chart) {
    chart.renderer.label(
      'This label is added in the callback ' + Highcharts.version + " " + _.round(4.006, 2),
      100,
      100
    )
    .attr({
      fill : '#90ed7d',
      padding: 10,
      r: 10,
      zIndex: 10
    })
    .css({
      color: 'black',
      width: '100px'
    })
    .add();
  };

  // NOTE: It should be placed in a separated file or be stringified in order it to work
  MyChart.prototype.onChartLoad = function (chart) {
    if (this.enableLabel && this.options.showLabel) {
      this.addLabel(chart);
    }
  };

  const myChart = new MyChart({
    seriesA: [1, 3, 2, 4, 3],
    seriesB: [5, 3, 4, 2, 5]
  }, {
    useSpline: true,
    showLabel: true
  });

  const fileName = 'outfile.png';

  const exportSettings = {
    export: {
      type: 'png',
      outfile: fileName,
      options: myChart.getChartOptions()
    },
    customCode: {
      allowCodeExecution: true,
      allowFileResources: true,
      callback: `function(chart) {
        chart.renderer.label(
          'This label is added in the callback ' + Highcharts.version + " " + _.round(4.006, 2),
          100,
          100
        )
        .attr({
          fill : '#90ed7d',
          padding: 10,
          r: 10,
          zIndex: 10
        })
        .css({
          color: 'black',
          width: '100px'
        })
        .add();
      }`,
      customCode: `function() {
        Highcharts.setOptions({
          chart: {
            events: {
              render: function () {
                this.renderer.circle(100, 100, 50).attr({
                  fill: 'red',
                  stroke: 'black',
                  'stroke-width': 1
                }).add();
              }
            }
          },
          title: {
            text: 'Changed through custom code ' + Highcharts.version + ' ' + _.round(4.006, 2)
          }
        });
      }`
    }
  };

  // Get the default options and merge them with above user options
  const options = puppeteerHighchartsExportServer.getDefaultOptions(exportSettings);

  // Initi pool with the final options
  await puppeteerHighchartsExportServer.initPool(options);

  // Run the export proccess
  puppeteerHighchartsExportServer.startExport(options, (info, error) => {
    if (error) {
      logger.error('error generating chart:', error);
      res.sendStatus(400);
      return;
    }

    // Simply display the base64 representation of a chart
    console.log(info.data);
  });
};

Once again, thank you for your insights and suggestions. We'll constantly upgrade, fix and optimize Puppeteer branch, so any further suggestions are welcome. And ofc. In case of any further questions, feel free to contact us.

[drmrbrewer]

Thanks for the reply, @PaulDalek !

To include the lodash script, you should place it in the scripts.value array, in the ./lib/schemas/config.js file.

You will see that I already tried this approach in my updated code above. Did I do it wrong there? It doesn't seem to work. If there is no way to load dependencies like this (easily, via the config options) it would make the custom code feature rather limited IMHO because in many cases we will need to rely on other libraries.

Here is the newest commit for you: 70e1d04.

But this commit isn't in the puppeteer branch itself? So far as I can see, the latest commit in the puppeteer branch is this one?

The reason why function related options can only be passed that way is because later in the code they are inserted directly in the stringified template

So it will never be possible with this library to define a callback or customCode function directly (like I suggest in my first post above) e.g. with reference to other object definitions in the same code as the highcharts module is being used? i.e. like so:

callback: function (chart) {
    myChart.onChartLoad(chart);
}

If this won't be possible, I think it's a problem for me using this as a node module (and possibly for many others in all but the simplest of use cases), and maybe I need to use my own implementation.

I did already create a custom node implementation based on puppeteer (it's more like just using puppeteer to take a screen capture of my chart hosted on a remote host... could be any web page really but in my case it's a custom chart generated based on options passed to it via the URL)... it works fine... except that it seems to use more resources (RAM and CPU) than my equivalent native version based on phantomjs, and I hoped that a more native puppeteer implementation like yours would somehow be more efficient. But not being able to use callback or customCode functions directly (only stringified or in a separate file) is a stumbling block.

[PaulDalek]

Hi @drmrbrewer

But this commit isn't in the puppeteer branch itself?

Yes, this commit is on another branch, not yet merged into the Puppeteer branch, although it should be soon.

You will see that I already tried this approach in my updated code above.

I’ve tried your approach and it worked for me. The changes regarding adding the function getDefaultOptions that I mentioned previously are currently in a separate branch. I'll try to make this branch be reviewed and merged soon.

So it will never be possible with this library to define a callback or customCode function directly

We didn't change that behavior compared to the old solution (based on the PhantomJS). We can take a look at it and consider if we can make it work for a node module usage.

Except that it seems to use more resources (RAM and CPU) than my equivalent native version based on phantomjs, and I hoped that a more native puppeteer implementation like yours would somehow be more efficient.

Yes, to be honest it is a challenge for us too now, as the Puppeteer based solution turned out to be more demanding than the PhantomJS one. Right now it is not very performant. There's definitely still room for improvement here and we will delve into optimizing it.

[drmrbrewer]

We can take a look at it and consider if we can make it work for a node module usage.

I think that this would (if at all possible) make the library far more powerful and flexible when used as a node module. Thanks for considering.

[charles-at-stack]

For what it's worth, we got the puppeteer branch installed with npm install highcharts/node-export-server#enhancement/puppeteer. That should fetch the latest commit on the enhancement/puppeteer branch.

[drmrbrewer]

@ff0041 thanks, very helpful!

[drmrbrewer]

It seems that the sun icon is blocked by the not same origin error even in the official demo (ERR_BLOCKED_BY_RESPONSE.NotSameOrigin).

@PaulDalek what official demo is that? I'm trying to reproduce this problem in a simple demo, so that I can either fix it or report it, because I'm seeing it in my current phantomjs implementation for some symbols which I'm fetching from highcharts. I wonder if it relates to the new bot protection that seems to be in place across highcharts.com?

[ndubel]

#391 (comment)
Hello! I've tried this example and got:

Mon May 22 2023 12:33:25 GMT+0000 [notice] - [cache] Dependency cache is up to date, proceeding.
Mon May 22 2023 12:33:25 GMT+0000 [verbose] - [cache] writing new manifest
Mon May 22 2023 12:33:25 GMT+0000 [notice] - [browser] attempting to get a browser instance (try 0)
Mon May 22 2023 12:33:26 GMT+0000 [notice] - [browser] failed: Error: Failed to launch the browser process! undefined
[1023109:1023121:0522/123325.867118:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1023109:1023125:0522/123325.872154:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1023109:1023125:0522/123325.872215:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1023109:1023121:0522/123325.874208:ERROR:bus.cc(399)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[1023109:1023121:0522/123325.874241:ERROR:bus.cc(399)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[1023109:1023109:0522/123326.033232:ERROR:process_singleton_posix.cc(334)] Failed to create /var/www/node/xxx/test/tmp/SingletonLock: File exists (17)


TROUBLESHOOTING: https://pptr.dev/troubleshooting

    at ChildProcess.onClose (file:///var/www/node/xxx/test/node_modules/@puppeteer/browsers/lib/esm/launch.js:253:24)
    at ChildProcess.emit (node:events:525:35)
    at ChildProcess._handle.onexit (node:internal/child_process:291:12)

Any ideas?

[jszuminski]

It has been solved with this PR: #408

Now, using highcharts-export-server as a node module should be simple.

The readme section has been updated to reflect those changes: https://github.com/highcharts/node-export-server/tree/enhancement/puppeteer#using-as-a-nodejs-module

For now, these changes are available on GitHub, but they will be available on NPM after the next minor release.

I'm closing this issue but please feel free to ask in case you find anything unclear.

[ronenl]

I have the same problem as @ndubel
When running the process for the first time it works without any issue, but after the "tmp" folder was created, running the process again will throw the error:

Mon Oct 02 2023 11:15:18 GMT+0300 [notice] - [browser] attempting to get a browser instance (try 22)
Mon Oct 02 2023 11:15:19 GMT+0300 [notice] - [browser] failed: Error: Failed to launch the browser process! undefined
[78460:259:1002/111519.209811:ERROR:process_singleton_posix.cc(334)] Failed to create /Users/ronen/examples/react-server-pdf-poc/tmp/SingletonLock: File exists (17)

The only way to WA the issue, is to manually delete the tmp folder and run the process again

[jszuminski]

@ronenl did you try with the latest version of the branch?

[ronenl]

@jakubSzuminski Yes I did, but getting the same problem

[jszuminski]

@ronenl it seems to be unrelated with this issue, but more with this one: #412

This issue has been prioritized and is no. 1 in our backlog. Your hint about the tmp folder is a huge help. Please track the other issue's progress on GH and I'll let you know when we fix it.