Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
771 lines (575 sloc) 18.8 KB

Method - Bundle

This method allows to generate fewer requests than the "Request Observable API" method, but is more complicated since it requires to manage more files and concepts.

Diagram for the "Bundle" method

TL;DR

This tutorial gives details about every step. Go to Standalone App Notebook (and its blog post) or joyplot/ for the direct solution.

Pros

  • generates fewer HTTP requests (the Observable notebook, the runtime.js Observable library and all the "dynamic dependencies" are bundled in a minified public/main.min.js JavaScript file)
  • allows to render only some cells of the notebook
  • supports old browser thanks to Babel (does not require browser compatibility with ES modules)
  • CSV data is downloaded from GitHub, so that the application runs offline.

Cons

  • complex: requires knowledge about npm, node.js, ES modules, now.sh, rollup, babel, terser...

Tutorial

This tutorial will lead to successive versions of the standalone webpage, with additional features in each step.

Step 1 - setup a npm package

The first step gives code similar to the "request Observable API" method, but with some of the modules retrieved locally (the notebooks).

  • Install node.js and npm

  • Create a new npm project:

    mkdir joyplot
    cd joyplot
    npm init
    package name: (joyplot)
    version: (1.0.0)
    description:
    entry point: (index.js)
    test command:
    git repository:
    keywords:
    author:
    license: (ISC)
    About to write to /[...]/joyplot/package.json:
    
    {
      "name": "joyplot",
      "version": "1.0.0",
      "description": "",
      "main": "index.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
      },
      "author": "",
      "license": "ISC"
    }
    
    Is this OK? (yes)
    
  • Install the notebook as a development dependency (the URL is obtained on the notebook page, clicking on "…" and then on "Download tarball"):

    mkdir -p vendor/
    curl -o vendor/notebook.tgz https://api.observablehq.com/@mbostock/psr-b1919-21.tgz?v=3
    npm install --save-optional aaa_notebook@file:vendor/notebook.tgz

    Note that it's exactly the same as the "default Observable export" method: the same .tgz file is downloaded.

    Note that the npm install --save-optional observable_notebook@file:vendor/notebook.tgz command has created the node_modules/ directory and extracted the tgz file to node_modules/aaa_notebook.

    Note also that it created

    "optionalDependencies": {
      "aaa_notebook": "file:vendor/notebook.tgz"
    }

    in package.json. This way, if npm install is launched in a fresh version of this repository, the notebook package will not be installed. It's OK because the vendor/notebook.tgz would not be available at this stage. npm run notebook would then install it.

  • Let's add a script to automate this each time we want to update the notebook to a new version:

    "scripts": {
      "notebook": "rm -f vendor/notebook.tgz && mkdir -p vendor && curl -o vendor/notebook.tgz $npm_package_custom_notebook && npm install --save-dev aaa_notebook@file:vendor/notebook.tgz",
      "test": "echo \"Error: no test specified\" && exit 1"
    },
    "custom": {
      "notebook": "https://api.observablehq.com/@mbostock/psr-b1919-21.tgz?v=3"
    },

    try it with:

    npm run notebook
  • Install the Observable runtime module:

    npm install --save-dev @observablehq/runtime@4
  • Create src/index.html file with the following content:

    <!DOCTYPE html>
    <html lang="en">
      <head>
        <!-- Minimal HTML head elements -->
        <meta charset="utf-8" />
        <title>PSR B1919+21</title>
      </head>
      <body>
        <!-- Title of the page -->
        <h1>PSR B1919+21</h1>
    
        <!-- Empty placeholders -->
        <div id="joyplot"></div>
    
        <!-- JavaScript code to fill the empty placeholders with notebook cells
             Note that the script is not vanilla JavaScript but an ES module
             (type="module")
          -->
        <script type="module" src="./main.js"></script>
      </body>
    </html>
  • create the src/main.js file with the following content:

    // Import Observable notebook
    // Note the relative path via ./node_modules - it's not optimal and will
    // be improved in the next steps
    import notebook from './node_modules/@mbostock/psr-b1919-21/@mbostock/psr-b1919-21.js';
    
    // Import Observable library
    // Same observation
    import {
      Runtime,
      Inspector,
    } from './node_modules/@observablehq/runtime/dist/runtime.js';
    
    // Render selected notebook cells into DOM elements of this page
    const runtime = new Runtime();
    const main = runtime.module(notebook, name => {
      switch (name) {
        case 'chart':
          // render 'chart' notebook cell into <div id="joyplot"></div>
          return new Inspector(document.querySelector('#joyplot'));
          break;
      }
    });
  • install a webserver:

    npm install --save-dev http-server@0

    add the script to package.json:

    "serve": "http-server ./",

    and serve with

    npm run serve

    Open http://127.0.0.1:8080/src/.

Step 2 - bundle JS modules in one file

In step 1, 3 local requests are made for main.js, psr-b1919-21.js and runtime.js. Moreover, the web server needs to be serving the "technical" node_modules/ directory, that normally should be reserved for development and build.

Instead, we will generate a single ES module, concatenating all the modules using rollup.js.

  • the sources (src/ directory) will be compiled to a new directory (public/):

    mkdir public
  • install rollup as a development dependency:

    npm install --save-dev rollup@1
  • install rollup-plugin-node-resolve to tell rollup to search the modules inside the node_modules/ directory:

    npm install --save-dev rollup-plugin-node-resolve@5
  • create the rollup.config.js configuration file for rollup:

    import * as meta from './package.json';
    import resolve from 'rollup-plugin-node-resolve';
    
    const onwarn = function(warning, warn) {
      if (warning.code === 'CIRCULAR_DEPENDENCY') {
        return;
      }
      warn(warning);
    };
    const output = {
      file: `public/main.js`,
      name: '${meta.name}',
      format: 'iife',
      indent: false,
      extend: true,
      banner: `// ${meta.homepage} v${
        meta.version
      } Copyright ${new Date().getFullYear()} ${meta.author.name}`,
    };
    
    export default {
      input: 'src/main.js',
      onwarn: onwarn,
      output: output,
      plugins: [resolve()],
    };

    It tells rollup to take the src/main.js file as the source, to find the dependencies (import statements), and to concatenate them all into public/main.js.

  • add the following script to package.json file to builds the public/main.js file and copy index.html:

    "build": "rollup -c && cp src/index.html public/",
  • adapt the src/main.js file to remove the relative paths since rollup-plugin-node-resolve will take care of finding the modules:

    // Import Observable notebook
    import notebook from 'aaa_notebook';
    
    // Import Observable library
    import {Runtime, Inspector} from '@observablehq/runtime';
  • build the file:

    npm run build
  • configure the webserver to use public/ instead of src/:

    "serve": "http-server public/",

    serve with

    npm run serve

    and access with http://127.0.0.1:8080/.

Step 3 - Bundle dynamically loaded modules

In step 2, the bundled module helps to save three requests, but it doesn't help with the remaining requests done at runtime to other dependencies (here d3.min.js, that is downloaded from jsdelivr.net). In this step, let's declare these dependencies statically (via import statements) and embed them into the bundle.

  • exploring the notebook, we see that it requires "d3@5". To find all the required modules, look for the require() function in the notebook module:

    $ grep -r "require(" node_modules/aaa_notebook
    ./node_modules/aaa_notebook/@mbostock/psr-b1919-21.js:require("d3@5")

    The "requests" tab in the browser developer tools will also show the modules fetched from jsdelivr.net.

  • install this dependency as a development dependency. Take care of installing the exact same version (see npm-semver for more details on how to specify the npm packages versions):

    npm install --save-dev d3@5.x
  • import it in main.js:

    // Import d3
    import * as d3 from 'd3';
  • now install d3-require. It will be used to override the normal way Observable's runtime dynamically gets access to the dependencies:

    npm install --save-dev d3-require@1
  • import d3-require in src/main.js. Also import the Library object from @observablehq/runtime.

    // Import d3-require
    import {require} from 'd3-require';
    
    // Import Observable library
    import {Runtime, Inspector, Library} from '@observablehq/runtime';
  • override the resolve function in runtime, when the module to resolve is "d3@5", replacing

    // Render selected notebook cells into DOM elements of this page
    const runtime = new Runtime();

    by

    // Resolve "d3@5" module to current object `d3`
    // Be careful: the 'd3@5' alias must be *exactly* the same string as in your
    // notebook cell, `d3 = require("d3@5")` in our case. Setting 'd3' or
    // 'd3@5.11.0' would not work
    const customResolve = require.alias({'d3@5': d3}).resolve;
    
    // Render selected notebook cells into DOM elements of this page
    // Use the custom resolve function to load modules
    const runtime = new Runtime(new Library(customResolve));
  • build and launch with:

    npm run build && npm run serve

    Tip: to build automatically when running the server, add the following script to package.json:

    "preserve": "npm run build",

    and just call:

    npm run serve
  • taking advantage of the modules, let's split the src/main.js:

    • src/main.js

      // Import Observable notebook
      import notebook from 'aaa_notebook';
      import {render} from './render';
      
      render(notebook);
    • src/render.js

      // Import Observable library
      import {Runtime, Inspector, Library} from '@observablehq/runtime';
      import {customResolve} from './customResolve';
      
      // Load modules locally
      const runtime = new Runtime(new Library(customResolve));
      
      // Render selected notebook cells into DOM elements of this page
      export function render(notebook) {
        runtime.module(notebook, name => {
          switch (name) {
            case 'chart':
              // render 'chart' notebook cell into <div id="joyplot"></div>
              return new Inspector(document.querySelector('#joyplot'));
              break;
          }
        });
      }
    • src/customResolve.js

      import {require} from 'd3-require';
      import * as d3 from 'd3';
      
      // Resolve "d3@5" module to current object `d3`
      // Be careful: the 'd3@5' alias must be *exactly* the same string as in your
      // notebook cell, `d3 = require("d3@5")` in our case. Setting 'd3' or
      // 'd3@5.11.0' would not work
      export const customResolve = require.alias({
        'd3@5': d3,
      }).resolve;

Step 4 - serve the data locally

In the previous step, everything is loading locally, except the pulsar.csv file that is accessed from gist.githubusercontent.com. Let's download the data and serve them locally (this part has been developed first in Fil/SphericalContoursStandalone and added later to this tutorial for completeness).

  • create a directory for the local data:

    mkdir -p public/data
  • download the data:

    curl -o public/data/pulsar.csv https://gist.githubusercontent.com/borgar/31c1e476b8e92a11d7e9/raw/0fae97dab6830ecee185a63c1cee0008f6778ff6/pulsar.csv
  • edit the code to make the fetch function download the data locally:

    • src/render.js: add

      import {customFetch} from './customFetch';
      
      // Fetch data locally
      fetch = customFetch;
      
    • src/customFetch.js:

      const fetchAlias = {
        'https://gist.githubusercontent.com/borgar/31c1e476b8e92a11d7e9/raw/0fae97dab6830ecee185a63c1cee0008f6778ff6/pulsar.csv':
          'data/pulsar.csv',
      };
      
      // intercept and reroute calls
      const _fetch = fetch;
      export function customFetch() {
        const a = arguments;
        if (a[0] in fetchAlias) a[0] = fetchAlias[a[0]];
        return _fetch(...a);
      }
  • check that the data is loaded locally:

    npm run serve

In order to be more generic, let's define the map between URL and local file only once:

  • create src/fetchAlias.json

    {
      "https://gist.githubusercontent.com/borgar/31c1e476b8e92a11d7e9/raw/0fae97dab6830ecee185a63c1cee0008f6778ff6/pulsar.csv": "data/pulsar.csv"
    }
  • edit src/customFetch.js

    import {default as fetchAlias} from './fetchAlias.json';
    
    // intercept and reroute calls
    const _fetch = fetch;
    export function customFetch() {
      const a = arguments;
      if (a[0] in fetchAlias) a[0] = fetchAlias[a[0]];
      return _fetch(...a);
    }
  • install rollup-plugin-json to allow this way to import json directly:

    npm install --save-dev rollup-plugin-json@4
  • adapt rollup.config.js

    import json from 'rollup-plugin-json'
    
    plugins: [resolve(), json()],
  • add a node utility to download the data locally:

    • create bin/download-data.js

      #! /usr/bin/env node
      const request = require('request');
      const fs = require('fs');
      const fetchAlias = require('../src/fetchAlias.json');
      const DIR = './public';
      
      for (const url of Object.keys(fetchAlias)) {
        const filename = `${DIR}/${fetchAlias[url]}`;
        console.warn(`download ${filename} from ${url}`);
        request(url).pipe(fs.createWriteStream(filename));
      }
    • install the required packages:

      npm install --save-dev request@2
    • register the script as a binary in package.json

      "bin": {
        "download-data": "bin/download-data.js"
      },
    • add a script in package.json to download or update the data

      "data": "rm -rf public/data && mkdir -p public/data && node bin/download-data.js",
    • test it with:

      npm run data
      npm run serve

Step 5 - support old browsers

As all the required modules have been bundled into one file (public/main.js), we are able to apply transpilation to it, so that old browsers will be able to run the JavaScript as well.

  • install babel and rollup-plugin-babel plugin

    npm install --save-dev @babel/core@7 rollup-plugin-babel@4
  • adapt rollup.config.js:

    //...
    import babel from 'rollup-plugin-babel';
    //...
      plugins: [resolve(), json(), babel()],
    //...
  • adapt public/index.html to remove type="module", since the code has been transpiled to non-module format

    <!-- JavaScript code to fill the empty placeholders with notebook cells -->
    <script src="./main.js"></script>
  • check

    npm run serve

Step 6 - minimize JavaScript code

Additionally we can minimize the size of the JS bundle:

  • install the terser plugin

    npm install --save-dev rollup-plugin-terser@5
  • adapt rollup.config.js:

    import {terser} from 'rollup-plugin-terser';
    // ...
      output: {...output, file: `public/main.min.js`, sourcemap: true},
      plugins: [resolve(), json(), babel(), terser()],
    // ...

    Note that we changed the name of the generated js file, and that we instruct rollup to produce the sourcemap in order to allow debugging the minified code.

  • adapt public/index.html to change the JavaScript filename

    <script src="./main.min.js"></script>
  • build and launch with:

    rm public/main.js # Obsolete file
    npm run serve

The file size is 292KB, whereas the original bundle was 592KB and the transpiled one (with polyfills to support old browsers) was 596KB.

Step 7 - publish online

In the previous step, the final files are made available in the public/ directory, but there is still a need to configure hosting to publish them online. There are various solutions to help publish from the cli, let's see one of them: now.sh.

Note that here we don't send the public/ directory to now.sh, but instead we tell now.sh to get the notebook, the data and to build the files itself.

  • install "now" as a development dependency:

    npm install --save-dev now@16
  • add a npm script to package.json to deploy to the now.sh hosting infrastructure:

    "deploy": "now",
  • in package.json, add node 10.x as the minimal node version (required by now@16):

    "engines": {
      "node": ">=10.x"
    },
  • create three files dedicated to now.sh:

    • now.json

      {
        "builds": [
          {
            "src": "now.sh",
            "use": "@now/static-build",
            "config": {"distDir": "public"}
          }
        ]
      }
    • now.sh

      #!/bin/bash
      
      npm run notebook && npm install && npm run data && npm run build
    • .nowignore

      node_modules/
      public/
      vendor/
      package-lock.json
      yarn.lock
      

The name of the project will be used to construct the URL on now.sh — if you forked an existing project, make sure that you changed the name (the first line in package.json) accordingly.

Note that it works even if we never build the files locally:

  • add a clear script to remove the temporary files:

    "clean": "rm -rf node_modules public vendor",
  • test the now.sh deployment on a fresh copy (we use npx now to avoid having node_modules, public, or vendor directories in local--note that it requires to have now installed globally):

    npm run clean
    npx now
    

Final code

The code can be found in joyplot/ directory.

You can’t perform that action at this time.