Not clear how to use ArrayBuffer backend with Emscripten FS

[vitonsky]

Hi, I want to mount ArrayBuffer FS on Emscripten FS. How to do it?

My use case

I use https://pglite.dev/ and want to keep FS in RAM, but dump this FS as ArrayBuffer once changes occurs.

PGLite have method https://pglite.dev/docs/api#dumpdatadir that even with no compression takes 100ms and this is too long time, since data dump may be called frequently.

Since PGLite uses a MEMFS i think the problem is they iterate whole FS content to build archive every time we call dump.

As potential solution I see a FS that will store data in single ArrayBuffer and will let us dump this buffer immediately (only time to copy buffer).

It looks your package implements exactly I need

import { configureSingle, fs, SingleBuffer } from "@zenfs/core";

const sharedArrayBuffer = new ArrayBuffer(100000);

await configureSingle({
	backend: SingleBuffer,
	buffer: sharedArrayBuffer,
});

fs.writeFileSync('/test.txt', 'You can do this anywhere, including browsers!');

The problem

So my goal is to

• Create SingleBuffer FS
• Mount this FS as emscripten FS at specific path /tmp/pglite/base
• Ensure that if i save buffer in FS, and then load it in another session and will init a SingleBuffer FS with that, i will have the same content (paths and files) with the same properties (time and access flags)

I just read the docs of https://www.npmjs.com/package/@zenfs/emscripten and docs of core and can't understand how to do it.

I see how to create SingleBuffer FS, and I see i can do like that

await configureSingle({ backend: Emscripten, FS: FS });

but it's not clear how to mount SingleBuffer FS on EMFS at specific path.

Maybe it's just problem with docs. Could you elaborate it

[james-pre]

Hey @vitonsky, thanks for reaching out.

In this case you want Emscripten to use ZenFS, not the other way around.

You should be able to use Emscripten's NODEFS/NODERAWFS and plug ZenFS in. You can configure ZenFS to use the backends you want:

import { configure, SingleBuffer, InMemory } from '@zenfs/core' 

await configure({
	mounts: {
		'/': {
			backend: SingleBuffer,
			buffer: new SharedArrayBuffer(1_000_000),
		},
		'/tmp': InMemory,
	},
});

Then, tell Emscripten to use NODERAWFS so you don't need to touch the original native code:

emcc ... -lnodefs.js -s NODERAWFS=1

Alternatively you can use regular NODEFS and mount it using Emscripten's FS.mount API.

I also looked into the PGLite docs and saw that you can just use node:fs (here). While you can alias fs to @zenfs/core, it unfortunatly doesn't seem possible to alias path to @zenfs/core/path.

[vitonsky]

@james-pre i does it like that

import { MemoryFS, PGlite } from '@electric-sql/pglite';
import { BaseFilesystem, PGDATA } from '@electric-sql/pglite/dist/fs/base';
import { c } from '@electric-sql/pglite/dist/pglite-CyDq4d4K';
import { configure, fs, SingleBuffer } from '@zenfs/core';
import EmscriptenPlugin from '@zenfs/emscripten/dist/plugin';

class CustomFS1 extends MemoryFS {
	init(
		pg: PGlite,
		emscriptenOptions: Partial<c>,
	): Promise<{ emscriptenOpts: Partial<c> }> {
		emscriptenOptions.preRun?.push((mod) => {
			const ZenFS = new EmscriptenPlugin(fs, mod.FS, mod.PATH);
			mod.FS.mkdir(PGDATA);
			mod.FS.mount(ZenFS, { root: '/' }, PGDATA);
		});
		return super.init(pg, emscriptenOptions);
	}
}

const sharedArrayBuffer = new ArrayBuffer(52_428_800);

await configure({
	mounts: {
		'/': {
			backend: SingleBuffer,
			buffer: sharedArrayBuffer,
		},
	},
});

const pg = new PGlite({
	fs: new CustomFS1(),
});

await pg.waitReady;

I got error

TypeError: Cannot read properties of undefined (reading 'join2')
 ❯ Object.lookup node_modules/@zenfs/emscripten/dist/plugin.js:69:40
     67|             },
     68|             lookup: (parent, name) => {
     69|                 const path = this.path.join2(this.realPath(parent), name);
       |                                        ^
     70|                 const mode = this.getMode(path);
     71|                 return this.createNode(parent, name, mode);

The mod.PATH is undefined

[james-pre]

I haven't used pglite before so I didn't know if it also passed the Emscripten PATH module— the answer is no. Just replace mod.PATH with a reference to Emscripten's PATH. It is probably a global. Otherwise you can try adding the functions yourself; the interface is here.

[james-pre]

Hey @vitonsky, just following up. Is everything working correctly or are you still experiencing issues? If you are experiencing issues you can try @zenfs/emscripten v1.0.5 and see if that fixes the problem.

[vitonsky]

@james-pre it looks the problem currently on pglite side, I've file an issue to track this problem electric-sql/pglite#810