Explorations: Stream API

[adamziel]

⚠️ Do not merge ⚠️ I use this large PR to explore, but I'd rather merge these changes a series of smaller PRs that are easier to discuss and review:

• Blueprints internals – split the common.ts file into smaller files #898
• PHP.wasm: Report both the source and target path on failure #896
• Move the stream-compression package from wp-playground to php-wasm #895
• Polyfills for Node.js v18 and JSDOM runtimes #875
• Stream-based zip encoder and decoder #880
• CI: Check if both CJS and ESM builds succeeded #876
• Streams: Migrate Blueprints, Website, GitHub integration to Streams #877

Description

Implements stream-based ZIP encodes and decoder using the CompressionStream and DecompressionStream class.

Here's what we get:

• Native ZIP support without having to rely on PHP's ZipArchive
• Download and unzip WordPress plugins at the same time. Before this PR we had to download the entire bundle, pass it to PHP, run PHP code, and only then the file would be unzipped.
• Partial download of large zip files.

To that last point:

ZIP as a remote, virtual filesystem

This change enables fast previewing of even 10GB-large zipped exports via partial downloads.

Imagine previewing a large site export with many photos and videos. The decodeRemoteZip function knows how to request just the list of files first, filter out the large ones, and then issue multiple fetch() requests to download the rest.

Effectively, we would only download ~5MB - 10MB of data for the initial preview, and then only download these larger assets once they're needed.

Technical details

Here's a few interesting functions shipped by this PR. Note the links point to a specific commit and may get outdated:

• nextZipEntry() that decodes a zipped file
• decodeRemoteZip() lists ZIP files in a remote archive, filters them, and then downloads just the subset of bytes we need to get those files
• encodeZip() turns a stream of File objects into a zip archive (as stream of bytes)

Remaining work

There's a few more things to do here, but I still wanted to get some reviews in before spending the time on these just in case the API would substantially change:

• Add unit tests.
• Solve conflicts
• Get the CI checks to pass.
• Test in Safari where the support for streams seems to be limited somehow (MDN shows red X-es on a few pages, let's investigate this)
• This PR updates Blueprints and the GitHub integration to demonstrate the impact on the codebase and give the reviewers more context. These changes will be offloaded to a series of follow-up PRs before this one is merged.
• Merge!
• Release new npm packages.
• Refactor progress monitoring to use ReadableStream.tee() instead of the workaround we use now – once https://bugs.chromium.org/p/chromium/issues/detail?id=1512548 is fixed in chromium.

API changes

Breaking changes

This PR isn't a breaking change yet. One of the follow-up PRs will very likely propose some breaking changes, but this one only extends the available API.

Without this PR

Without this PR, unzipping a file requires writing it to Playground, calling PHP's unzip, and removing the temporary zip file:

const response = await fetch(remoteUrl);
// Download the entire byte array first
const bytes = new Uint8Array(await response.arrayBuffer());
// Copy those bytes into Playground memory
await writeFile(playground, {
	path: tmpZipPath,
	data: zipFile,
});
// Run PHP code and use `ZipArray` via unzip()
await unzip(playground, {
	zipPath: tmpZipPath,
	extractToPath: targetPath,
});
// Only now is the ZIP file extracted.
// We still need to clean up the temporary file:
await playground.unlink(tmpZipPath);

With this PR

With this PR, unzipping works like this

const response = await fetch(remoteUrl);
// We can now unzip as we stream response bytes
decodeZip( response.body )
	// We also write the stream of unzipped files to PHP as new entries become available
	.pipeTo( streamWriteToPhp( playground, targetPath ) )

More examples

Here's what else the streaming API unlocks. Not all of these functions are shipped here, but they are quite easy to implement:

// In the browser, fetch a zip file:
(await fetch(url))
	.body
	.pipeThrough(decodeZip())
	.pipeTo(streamWriteToPhp(php, pluginsDirectory))

// In the browser, install from a VFS directory:
iteratorToStream(iteratePhpFiles(path))
	.pipeTo(streamWriteToPhp(php, pluginsDirectory))

// In the browser, install from a .zip inside VFS:
streamReadPhpFile(php, path)
	.pipeThrough(decodeZip())
	.pipeTo(streamWriteToPhp(php, pluginsDirectory))

// Funny way to do a recursive copy
iteratorToStream(iteratePhpFiles(php, fromPath))
	.pipeTo(streamWriteToPhp(php, toPath))

// Process a doubly zipped artifact from GitHub CI
(await fetch(artifactUrl))
	.body
	.pipeThrough(decodeZip())
	.pipeThrough(readBody())
	.pipeThrough(decodeZip())
	.pipeTo(streamWriteToPhp(php, pluginsDirectory))

// Export Playground files as zip
iteratorToStream(iteratePhpFiles(php, fromPath))
	.pipeThrough(encodeZip())
	.pipeThrough(concatBytes())
	.pipeTo(downloadFile('playground.zip'))

// Export Playground files to OPFS
iteratorToStream(iteratePhpFiles(php, fromPath))
	.pipeTo(streamWriteToOpfs('/playground'))

// Compute changeset to export to GitHub
changeset(
	iterateGithubFiles(org, repo, branch, path),
	iteratePhpFiles(php, fromPath)
);

// Read a subdirectory from a GitHub repo
decodeRemoteZip(
	zipballUrl,
	({ path }) => path.startsWith("themes/adventurer")
)
	.pipeThrough(enterDirectory('themes/adventurer'))
	.pipeTo(streamWriteToPhp(php, joinPath(themesPath, 'adventurer')))

// Write a single file from the zip into a path in PHP
decodeRemoteZip(
	artifactUrl,
	({ path }) => path.startsWith("path/to/README.md")
)
	.pipeTo(streamWriteToPhp(php, '/wordpress'))

// In node.js, install a plugin from a disk directory
iteratorToStream(iteratePhpFiles(php, path))
	.pipeTo(streamWriteToPhp(php, pluginsDir))
;

Open questions

How can streams be expressed in Blueprints?

Solving this is out of scope for this PR, but it's one of the next problems to explore so we might as well use this PR as a space to contemplate. I'll move this into a separate issue once we solidify the shape of the API here and ship this PR.

URI notation

Perhaps the Blueprint steps could accept a protocol://URI notation? This is the option I'm leaning towards, unless a better alternative comes up.

Upsides:

• Succinct
• Somewhat familiar, e.g. git does git+https://git@...

Downsides:

• It still looks a bit foreign and requires some getting used to
• It cannot easily express nested resources like "A path inside a zip" or "A stream inside a zip"

{
	"steps": [
		{
			step: 'installPlugin',
			pluginDir: 'zipdir+plugin://hellodolly',
		},
		{
			step: 'installPlugin',
			pluginDir: 'zipdir+https://mysite.com/plugin.zip:hello-dolly/',
		}
	]
}

Object-based DSL

We can have unlimited flexibility with a custom DSL like below

Upsides:

• Flexibility

Downsides:

• Even more complex than the URI proposal. Requires training, looks foreign, noone would be able to write it without consulting the documentation.

{
	"steps": [
		{
			step: 'installPlugin',
			pluginFiles: {
				"resource": "unzip",
				"pathInside": "hello-dolly/",
				"zipFile": {
					"resource": "https",
					"url": "https://mysite.com/hello-dolly.zip"
				}
			}
		}
	]
}

Piping DSL

How about we mimic the JavaScript API using the JSON notation?

Upsides:

• Flexibility
• Less complex than the object-based DSL above
• Looks similar to the JavaScript counterpart

Downsides:

• We're essentially JavaScript and creating a micro-language in JSON. I don't want people to express their code in JSON, I want them to have a convenient tool that can easily express complex concepts.

{
	"steps": [
		{
			step: 'installPlugin',
			pluginFiles: { "pipe": [
				{ "fetch": "https://mysite.com/hello-dolly.zip" },
				"unzip",
				{ "enterDirectory": "hello-dolly/" }
			} ]
		}
	]
}

cc @dmsnell

[dmsnell]

Very nice to see the elimination of the intermediate unpacked folder. This should bring a nice improvement in the latency of installing ZIP files, leading to things finishing quite a bit faster during installs and boot.

[dmsnell]

it appears that Emscripten is the real culprit here, but filenames are not encoded. they are bytes separated by ASCII /. this will end up breaking filenames and inserting � in places.

[dmsnell]

curious: what happens if we attempt to read more bytes than are available?

[adamziel]

I removed the custom Stream class implementation. Right now we read n bytes using the native ReadableStreamBYOBReader class which sends a "read request" to the data source. Every data source is free to process it how it wants, but typically it will only the buffer up to Math.min(Buffer.size, numberOfAvailableDataBytes).

[adamziel]

Safari doesn't support BYOB streams which is a blocker here. Perhaps there's a way to polyfill them? I experimented with that at 2b0acd0#diff-888d17d202646c67c560411a8e150946f30fec742ff592a7915d6c8370ab030e

[adamziel]

Surfacing the performance concerns from #919 (comment).

Buffering the entire .zip file and passing it to PHP.wasm in one go takes around 300ms on my machine. However, using the browser streams and writing each file as it becomes available takes around 2s where:

• ~50% is spent decoding the ZIP file, using ArrayBuffers, ReadableStreams, and such
• ~50% is spent sending each decompressed file to the PHP worker via postMessage

There's potentially a lot of room for improvement, but I'm not too keen about tweaking this.

Tweaking the stream-based implementation would take a lot of time, and I'm not convinced about the benefits. The lower bound for the execution time seems to be set by the native version of libzip and libz. My intuition is that:

• JavaScript–based zip decoder is essentially a lot of overhead on top of that lower bound including executing JavaScript code, copying the data in JS VM more than we need to, marshalling and sending the data via postMessage many times over etc.
• WASM zip decoder is that lower bound times a factor like wasmSlowerThanNative (which could be nuanced, non-linear etc).

All of this is hand wavy and based on my intuition. I don't have any actual measurements and perhaps I could spend a dozen or more hours here and either prove or disprove those assumptions, but I think there's a more promising avenue to explore instead.

I wonder if we could stream all the downloaded bytes directly to WASM memory, and stream-handle them there. JavaScript would become an API layer over WASM operations, much like numpy in python. We wouldn't be passing data around, just orchestrating what happens on the lower level. The API could look like this:

php
    .acceptDataStream( fetch( pluginFileURL ).body )
    .unzip()
    .writeTo( "/wordpress/wp-content/plugins/gutenberg" );

[adamziel]

Blueprints v2 make integrating JavaScript ZIP streaming into Blueprint steps unnecessary as the data is processed using PHP streams instead of JavaScript ones. Let's still keep the stream processing package around, though, as it's independent and useful to have.