File System Access API

[gnodar01]

There are several motivating reasons to move to File System Access API as discussed in #276, under "Project File Storage and Retrieval". With respect to issue #270, there is an alternative solution. We can create a streaming download solution, without having to use the File System Access API.

big-json is a package meant for node. Many node-oriented packages can be run in the browser via polyfills. Webpack <= 4 used to provide those polyfills by default, but Webpack > 4 does not. You can mess with the webpack config file to override this default, but since we’re using Create-React-App, that usually means ejecting, which isn’t the end of the world, but I’d rather avoid it. Alternatively we could use react-app-rewired to tweak webpack config without ejecting. With that, we can install the necessary node polyfills for big-json, which are stream-browserify and assert, and process. In project root, we would create config-overrides.js with something like the following, telling webpack how to resolve the missing dependencies:

const webpack = require("webpack");
module.exports = function override(config) {
  const fallback = config.resolve.fallback || {};
  Object.assign(fallback, {
    stream: require.resolve("stream-browserify"),
    assert: require.resolve("assert"),
  });
  config.resolve.fallback = fallback;
  config.plugins = (config.plugins || []).concat([
    new webpack.ProvidePlugin({
      process: "process/browser",
      Buffer: ["buffer", "Buffer"],
    }),
  ]);
  config.module.rules.unshift({
    test: /\.m?js$/,
    resolve: {
      fullySpecified: false, // disable the behavior
    },
  });
  return config;
};

Then in package.json, we’d need to replace all instances of react-scripts with react-app-rewired.

This will achieve streaming JSON stringification and parsing. With that in place, we can no longer use file-saver, because blobs have a size limitation as well (500 mb - 2 GB), similar to size limits for JSON.stringify, and for more generally variables holding strings. We’d have to replace it with StreamSaver.

StreamSaver uses a MITM in order to install a service worker, which uses respondWith to act as a psuedo-server. The data stream is transfered to the service worker using postMessage, and then the worker creates a download link that we open, allowing us to stream to disk.

big-json.createStringifyStream returns a JsonStreamStringify object (json-stream-stringify package), which is a NodeJS read stream. StreamSaver however uses the web stream API, and provides a web writable stream on createWriteStream. We can't just call pipe on the node-oriented read stream to the web-oriented write stream. To convert the node read stream to a web read stream, we can use readable-stream-node-to-web . Then we can call pipeTo on the newly converted web read stream to the web write stream.

// in src/components/SaveProjectDialog/SaveProjectDialog.tsx -> SaveProjectDialog() -> onSaveProjectClick()

const body = {
  classifier: classifier,
  project: serializedProject,
  version: "0.0.0",
};

body.project.name = projectName;

const stringifyStream: JsonStreamStringify = json.createStringifyStream({
  body: body,
}); // NodeJS readable

const fileStream = streamSaver.createWriteStream("test_project.json");
const webReadableStream: ReadableStream = nodeToWebStream(fileStream);

webReadableStream
  .pipeTo(fileStream)
  .then(() => console.log("done writing"))
  .catch((err) => console.error(err));

An alternative is to eschew piping altogether, and use the data event handler on the node-oriented read stream in order to build an array of stringified JSON chunks. Once that array is built, we can construct a Blob out of it, and stream that to the web write stream provided by StreamSaver.

// in src/components/SaveProjectDialog/SaveProjectDialog.tsx -> SaveProjectDialog() -> onSaveProjectClick()

const body = {
  classifier: classifier,
  project: serializedProject,
  version: "0.0.0",
};

body.project.name = projectName;

const stringifyStream: JsonStreamStringify = json.createStringifyStream({
  body: body,
}); // NodeJS readable

const fileStream = streamSaver.createWriteStream("test_project.json");

const pieces: string[] = [];

stringifyStream.on("data", function (strChunk: string) {
  console.log(`Read ${strChunk.length} bytes of data...`);
  pieces.push(strChunk);
});

stringifyStream.on("end", () => {
  console.log("Reached end of Stringify stream");

  // const blob = new Blob(pieces, { type: "application/json;charset=utf-8" });
  const blob = new Blob(pieces);
  const readableStream = blob.stream() as unknown as ReadableStream;

  readableStream
    .pipeTo(fileStream)
    .then(() => console.log("success"))
    .catch((err) => console.error(err));
});

stringifyStream.on("close", () => {
  console.log("Closed Stringify stream");
});

stringifyStream.on("error", (err) => {
  console.log("Error on Stringify stream", err);
});

[gnodar01]

The solution above works, but is hacky. I also haven't put much effort into thinking how it works when reading a very large json project file. We would presumably need to stream on the read side as well, and I haven't explored all of the nuances of that.

In any case it's clumsy solution, and far from ideal. It doesn't give us "real" disk access. For example we can't read and write temporary files, and we certainly can't perform seek operations on an open file descriptor. For those cases, we could use something like bro-fs or browserfs ([docs here])(http://jvilk.com/browserfs/1.4.1/), or use LocalStorage/IndexedDB directly. However, eventually we would want to write the contents to disk, and we're back to the JSON too large problem. We also can't traverse a directory in order to read multiple files, along with their pathnames (although browserfs lets us use Dropbox as a backend, so maybe that would be a less than ideal solution).

Given that, I think we should eschew the above solution, or any like it, and make the definitive decision to move towards File System Access API, abandoning Firefox support as a consequence.

We can also submit a request to the Firefox team in the dim hopes that they see reason.

[james-pre]

Hi @gnodar01, I'm the maintainer of BrowserFS. I'd like to let you know that BrowserFS has been deprecated. You can read more about on the BrowserFS readme (https://github.com/jvilk/BrowserFS). ZenFS is an updated and active fork that I created.