Look into use of DataViews and ArrayBuffers for more efficient data send/recv

[pzwang]

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView

[almarklein]

What we need

First a definition: this is all about improving the support for typed homogeneous array data. Let's call this a typed-array, which means e.g. Float32Array in JS, and a Numpy array in Python.

We need a format:

• A format to serialize structured data, either to be send directly between client and server, or to export to HTML, as an alternative to json, but with support for typed-arrays.
• On the Python side, serialize numpy arrays directly, without turning into a list first.
• On the JS side, parse directly into e.g. Float32Array, without going to a normal array first.
• The array data should be stored in an efficient manner. Storing in a binary format is preferred to avoid base64 encoding.
• Compression would be nice, but is not trivial in JS, so probably not a key necessity.
• Need support for typed-arrays data of different dtypes (uint16, float32, float64, etc.)
• Dimensionality: JS only knows 1D arrays, but maybe we want to support nd arrays at some point (via some sort of numpy.js)? It would also make the format usable in other contexts (e.g. as a lightweight scientific data container, or Python-R communication).

Proposal

I can see three options now:

A: use JSON
We can write a Python and JS "extension" to store typed-arrays as a dict, e.g.: {"__array__": "LONG_STRING_OF_BASE64", "size": 512, "dtype": "float32"}. The __array__ is a special marker than can be recognised by a decoder so this field can be decoded as a typed array. The advantage is that we still use JSON which is what ppl are familiar to. The disadvantage is that its not very efficient since the arrays need to be encoded/decoded in base64, which takes time and space.

B: use UBJSON
We can (mis)use UBJSON's feature of efficiently storing homogeneous arrays to store typed-arrays. The advantage is that its binary and fast, using an existing protocol (but we do have to modify the parsers a bit). The disadvantage is that we're using UBJSON in a way that it's not really intended for. Also would be limited to 1D arrays.

C: roll our own format
We could specify a new format with full support for typed arrays (and giving up JSON compatibility). If we derive it from UBJSON (which has a very smart and simple spec IMO) it would not be a huge task. The advantage is that we get exactly what we need. The disadvantage is that we would add yet another competing format.

---

details below

---

Here's some "binary json" variants:

• ubjson is interesting in that it aims for compliance with JSON (whereas other JSON-ish formats introduce new stuff), yet they support efficient storage of typed homogeneous arrays, and with it allow storing binary data efficiently. The spec is also very simple to ease adoption. I don't think typed homogeneous arrays are parsed into JS typed arrays, but maybe we can write a JS parser that does this, and a Python parser that is numpy aware?
• msgpack is a binary JSON variant aimed to be very small. They allow binary data and also extensions (which makes less incompatible with JSON). No support for typed homogeneous arrays (but could be, with an extension). Indeed there is binary-pack, which adds support for "distinct string and binary types", cannot find any docs though.
• bson is another binary JSON format, that also has types like data, MD5 and more, making them not very compatible with JSON.
• smile another binary JSON, seems dated.
• bjson seems dead.

The problem with most of these is that we need something more than json. This makes these formats not very suitable, except perhaps ubjson, for which we could hijack the support for storing homogeneous arrays more efficiently as a means to store typed-arrays.

We could "extend" json by storing typed-arrays as base64 encoded strings. If we have a special encoder/decoder on both the JS and Python side, numpy arrays could transparently become typed JS arrays. See e.g. http://stackoverflow.com/a/24375113/2271927 and EJSON.

Other formats

• For reference, I once wrote the SSDF format, which has an implementation in Python and Matlab, that is similar to JSON (in its data types, and that its human readable), but also supports nd typed-arrays, which are zlib-compressed and then base64 encoded.
• XML: just kidding :)
• hdf5 is too much

Since we're interested in storing data, we quickly end up in the more scientific formats, which are generally rather complex. Should we consider coming up with something ourselves?

[almarklein]

cc @bryevdv we already had an issue for this. Will start looking into this next week.

[almarklein]

I looked into compression. Not necessary per see, but it would be nice to reduce the size of transfers and exported documents. Unfortunately, you'd need a 3d party library, and these are all pretty big.

Examples for compression schemes that are also built into Python:

• Pako is a zlib implementation in JS which is reportedly very fast. Comes in at a 5.5k sloc count though.
• js-deflate provides delate/inflate (i.e. zlib) compression at about 2k sloc.
• lzma is an LZMA (de)compressor at 2.6k sloc.
• here is a bz2 decompressor in 245 sloc, docs are very sparse, but it seems to have been the basis for many other bz2-related libs.

Other examples (would need something on the Python side):

• lz-string is a string compression lib (no typed arrays iiuc) provides LZW-ish compression at 485 sloc. Not an "official" format though.
  lz4.js provides lz4 compression at 6k sloc.
• minilzo implements LZO (de)compression in 560 sloc, based on minilzo.c which is written by the O in LZO.
• LZ77-kit provides LZ77 compression for various languages. LZ77 is a component of many other compression schemes, but on itself it is a very simple algorithm, very fast, and offering moderate compression.

[almarklein]

Ok, I think I've done enough googling and reading through format specs for now. I need some help moving further. In my first post of this issue I put together an overview and I propose three options. cc @bryevdv @pzwang

[bryevdv]

@almarklein An idea that has been bandied about was to remove actual data from ColumnDataSource, and instead have data sources be a lightweight objects that are configured with a "remote" actual data store. This could be a URL to a REST endpoint, or a Blaze server. Or, it could be a reference to a "local remote" data store that lives in the browser but is separate from all the other Bokeh models. I think if we did this kind of separation of the actual data payload from the lightweight data source model, it woudl allow all the normal bokeh objects to remain simple plain JSON, and then just the data columns could be transmitted separately in an enhanced JSON, or non-JSON format? Like the idea of making all the Bokeh models "lightweight" on its own, but it if would also ihelp preserve the simple JSON representation for the majority of things that would be another big point in favor for me. Thoughts @bokeh/core

[bryevdv]

Other comments: I'm OK with 1d only, if we want to do something simple like storing a simple shape in some conventional way, that would be ok too. Almost all the use-cases in bokeh are around tabular columns so that is what we should optimize for.

[almarklein]

If we separate the data, as you suggest, there is no longer a need for a structured data format, so we could probably do with something simpler in that case. I'm interested in hearing more about this ...

[bryevdv]

@almarklein we should probably have a call sometime soon. My current plan is to implement a binary protocol over web sockets. Doing this, it seems possible to send NumPy/Pandas data directly into a JS ArrayBuffer, which can have a type array view on it without any copying. I'd like to get more input for people (and possibly help as well)

[bryevdv]

To add a little more I intend to make the wire-protocol an implementation detail, so that we can change things later if we need to. For instance, msgpack has had some integration into blaze-server so it might make sense to look at. But for now I am just going to to arr.tobytes along with a head that has type/shape info, which works perfectly fine.

[almarklein]

If we're storing/sending the data separate, we only need to store a blob, a shape and a type, so a simple dedicated format is fine IMO. Msgpack seems overkill unless we want to send the data along with all the model stuff that we now send via json.

I assume we can use the same format to store data in static HTML (but base64 encoded)?

[damianavila]

Interesting stuff... @bryevdv, do you have a branch were you prototyped this?

[waylonflinn]

@bryevdv @almarklein I'm working out exactly the same data exchange problem for interoperability between numpy and weblas. I came to the same conclusion that you guys did: bytes with type and shape should be sufficient.

There aren't a ton of ways to do that, but I'd like to be compatible with you guys from the start. Do you have code or a simple spec you can share?