Implement in-memory worker file system

[jasnell]

So this is a fun one. In preparation for the implementation of node:fs (and eventually also the Web file system API), we need to implement a virtual file system for the worker. Consider this a WIP sketch for the time being. There are still a range of things to do and verify before this is ready.

There's quite a bit to unpack here so let's go through it a bit.

First, let's talk about the file system itself. Every worker instance will have its own root directory (/). In this root directory we will have at least three special directories, the "bundle" root, the "dev" root, and the "temp" root.

• The bundle root is where all of the modules that are included in the worker bundle will be accessible. Everything in this directory will be strictly read-only. The contents are populated by the worker configuration bundle. By default, the bundle root will be /bundle but this can be overridden.

• The dev root provides the equivalent to /dev/null, /dev/zero, /dev/full, and /dev/random.

• The temp root is where we will allow temporary files to be created. Everything in this directory is read-write but will be transient. By default, the temp root will be /tmp but this can also be overridden.

Let's imagine the following simple workerd configuration:

const helloWorld :Workerd.Worker = (
  modules = [
    (name = "worker",
     esModule = embed "worker.js"),
    (name = "foo", text = "Hello World!"),
  ],
  compatibilityDate = "2023-02-28",
);

Given this configuration, the worker fs will initially have the following structure:

/
├── bundle
│   ├── worker
│   └── foo
├── dev
│   ├── null
│   ├── zero
│   ├── full
│   └── random
└── tmp

We can then access the file system at the C++ level.

jsg::Lock& js = ..
KJ_IF_SOME(vfs, workerd::VirtualFileSystem::tryGetCurrent(js)) {
  auto dir = vfs.getRoot(js);
  KJ_IF_SOME(item, vfs.resolve(js, "file:///a/b/c"_url")) {
    KJ_SWITCH_ONEOF(item) {
     KJ_CASE_ONEOF(file, kj::Rc<workerd::File>) { /* ... */ }
     KJ_CASE_ONEOF(dir, kj::Rc<workerd::Directory>) { /* ... */ }
    }
  }
}

The temporary file directory is a bit special in that the contents are fully transient based on whether there is or is not an active IoContext. We use a special RAII TmpDirStoreScope scope to manage the contents of the temporary directory. For example,

    TempDirStoreScope temp_dir_scope;
    auto tmpFile = dir->tryOpen(js, path, workerd::FsType::FILE);
    KJ_ASSERT(tmpFile.write(js, 0, "Hello World!"_kjb) == 12);
    KJ_ASSERT_NONNULL(dir->stat(js, path));  // exists!
  }
  // The temp dir scope is destructed and the file is deleted
  KJ_ASSERT(dir->stat(js, path) == kj::none);
}

If there is an active IoContext, the temporary file wil be created within that IoContext's TmpDirStoreScope, and will be deleted when the IoContext is destructed. This allows us to have a single virtual file system whose temporary directories are either deleted immediately as soon as the execution scope is exited, or are specific to the IoContext and are deleted when the IoContext is destructed. This mechanism allows us to have multiple IoContexts active at the same time while still having a single virtual file system whose contents correctly reflect the current IoContext.

Temporary files can be created, copied, etc only within the temporary directory. All other directories are read-only.

When there is no active IoContext, all temporary files will have a timestamp set to the Unix epoch. When there is an active IoContext, the temporary files will acquire the current timestamp from the IoContext ensuring that the file system's view of time is consistent within a request.

The design here is intended to be extensible. We can add new root directories in the future with different semantics and implementations. For example, to support python workers we can introduce a new root directory that is backed, for instance, by a tar/zip file containing the python standard library, etc.

What isn't implemented yet? Great question! The following todos are remaining:

• Implementing "file descriptors". Currently the file system will return kj::none for all file descriptors. As a next step, the file system will implement a custom file descriptor counter that will ensure that all files are assigned a unique monotonically increasing file descriptor. When the counter reaches a maximum threshold of opened files, the worker will be condemned. We can also just track the number of active kj::Own<const kj::File> objects and use that internally for accounting but the node:fs impl does a lot with integer fds so we will need them at some point for that.

The implementation is split across several files/components:

worker-fs.h/c++ - These provide the main interfaces for the worker file system.

bundle-fs.h/c++ - These provide the interfaces for interfacing the workerd configuration with the worker file system. A similar interface will need to be provided for the internal repo since the two use different configuration schemas.

[jasnell]

Although this is marked draft, it is ready for an initial round of reviews.

[danlapid]

I'll stop reviewing at this point as there seems to be too much up in the air at this stage.
My general sentiment is that this is quite a bit overcomplicated by the API surface it seems to mimic.
I don't know if it's actually serving us.

Also, considering we've already implemented everything in C++ anyway and left very little for the imagination. I would love to see this interfaced as an API type for easy interfacing from different languages (Node,Python,Rust etc.)

[jasnell]

Ok... updated the PR with a few more touch ups and largely just to demonstrate that it works as expected, I through in an initial rough implementation of the Web FileSystem API (would need more tests, namely web platform forms) .. https://fs.spec.whatwg.org/

More importantly, this includes the cloudflare-internal:filesystem module that our node:fs implementation would build on top of. @anonrig .. this should replace the native bits in your other PR.

[github-actions]

The generated output of @cloudflare/workers-types has been changed by this PR. If this is intentional, run bazel build //types && rm -rf types/generated-snapshot && cp -r bazel-bin/types/definitions types/generated-snapshot to update the snapshot. Alternatively, you can download the full generated types:

Full Type Diff

diff -r types/generated-snapshot/experimental/index.d.ts bazel-bin/types/definitions/experimental/index.d.ts
259a260,265
>   FileSystemHandle: typeof FileSystemHandle;
>   FileSystemFileHandle: typeof FileSystemFileHandle;
>   FileSystemDirectoryHandle: typeof FileSystemDirectoryHandle;
>   FileSystemWritableFileStream: typeof FileSystemWritableFileStream;
>   FileSystemSyncAccessHandle: typeof FileSystemSyncAccessHandle;
>   StorageManager: typeof StorageManager;
471a478
>   readonly storage: StorageManager;
1380a1388,1529
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle)
>  */
> declare abstract class FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/kind) */
>   get kind(): string;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/name) */
>   get name(): string;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/isSameEntry) */
>   isSameEntry(other: FileSystemHandle): Promise<boolean>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle)
>  */
> declare abstract class FileSystemFileHandle extends FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/getFile) */
>   getFile(): Promise<File>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/createWritable) */
>   createWritable(
>     options?: FileSystemFileHandleFileSystemCreateWritableOptions,
>   ): Promise<FileSystemWritableFileStream>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle) */
>   createSyncAccessHandle(): Promise<FileSystemSyncAccessHandle>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle)
>  */
> declare abstract class FileSystemDirectoryHandle extends FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/getFileHandle) */
>   getFileHandle(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemGetFileOptions,
>   ): Promise<FileSystemFileHandle>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/getDirectoryHandle) */
>   getDirectoryHandle(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemGetDirectoryOptions,
>   ): Promise<FileSystemDirectoryHandle>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/removeEntry) */
>   removeEntry(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemRemoveOptions,
>   ): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/resolve) */
>   resolve(possibleDescendant: FileSystemHandle): Promise<string[]>;
>   entries(): IterableIterator<FileSystemDirectoryHandleEntryType>;
>   keys(): IterableIterator<string>;
>   values(): IterableIterator<FileSystemHandle>;
>   forEach(
>     callback: (
>       param0: string,
>       param1: FileSystemHandle,
>       param2: FileSystemDirectoryHandle,
>     ) => void,
>     thisArg?: any,
>   ): void;
>   [Symbol.iterator](): IterableIterator<FileSystemDirectoryHandleEntryType>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream)
>  */
> declare abstract class FileSystemWritableFileStream extends WritableStream {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/write) */
>   write(
>     data:
>       | Blob
>       | (ArrayBuffer | ArrayBufferView)
>       | string
>       | FileSystemWritableFileStreamWriteParams,
>   ): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/seek) */
>   seek(position: number): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/truncate) */
>   truncate(size: number): Promise<void>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle)
>  */
> declare abstract class FileSystemSyncAccessHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/read) */
>   read(
>     buffer: ArrayBuffer | ArrayBufferView,
>     options?: FileSystemSyncAccessHandleFileSystemReadWriteOptions,
>   ): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/write) */
>   write(
>     buffer: ArrayBuffer | ArrayBufferView,
>     options?: FileSystemSyncAccessHandleFileSystemReadWriteOptions,
>   ): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/truncate) */
>   truncate(newSize: number): void;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/getSize) */
>   getSize(): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/flush) */
>   flush(): void;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/close) */
>   close(): void;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/StorageManager)
>  */
> declare abstract class StorageManager {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/StorageManager/getDirectory) */
>   getDirectory(): Promise<FileSystemDirectoryHandle>;
> }
> interface FileSystemFileHandleFileSystemCreateWritableOptions {
>   keepExistingData: boolean;
> }
> interface FileSystemDirectoryHandleFileSystemGetFileOptions {
>   create: boolean;
> }
> interface FileSystemDirectoryHandleFileSystemGetDirectoryOptions {
>   create: boolean;
> }
> interface FileSystemDirectoryHandleFileSystemRemoveOptions {
>   recursive: boolean;
> }
> interface FileSystemSyncAccessHandleFileSystemReadWriteOptions {
>   at?: number;
> }
> interface FileSystemWritableFileStreamWriteParams {
>   type: string;
>   size?: number;
>   position?: number;
>   data?: Blob | (ArrayBuffer | ArrayBufferView) | string;
> }
> interface FileSystemDirectoryHandleEntryType {
>   key: string;
>   value: FileSystemHandle;
diff -r types/generated-snapshot/experimental/index.ts bazel-bin/types/definitions/experimental/index.ts
259a260,265
>   FileSystemHandle: typeof FileSystemHandle;
>   FileSystemFileHandle: typeof FileSystemFileHandle;
>   FileSystemDirectoryHandle: typeof FileSystemDirectoryHandle;
>   FileSystemWritableFileStream: typeof FileSystemWritableFileStream;
>   FileSystemSyncAccessHandle: typeof FileSystemSyncAccessHandle;
>   StorageManager: typeof StorageManager;
476a483
>   readonly storage: StorageManager;
1385a1393,1534
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle)
>  */
> export declare abstract class FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/kind) */
>   get kind(): string;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/name) */
>   get name(): string;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemHandle/isSameEntry) */
>   isSameEntry(other: FileSystemHandle): Promise<boolean>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle)
>  */
> export declare abstract class FileSystemFileHandle extends FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/getFile) */
>   getFile(): Promise<File>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/createWritable) */
>   createWritable(
>     options?: FileSystemFileHandleFileSystemCreateWritableOptions,
>   ): Promise<FileSystemWritableFileStream>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle) */
>   createSyncAccessHandle(): Promise<FileSystemSyncAccessHandle>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle)
>  */
> export declare abstract class FileSystemDirectoryHandle extends FileSystemHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/getFileHandle) */
>   getFileHandle(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemGetFileOptions,
>   ): Promise<FileSystemFileHandle>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/getDirectoryHandle) */
>   getDirectoryHandle(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemGetDirectoryOptions,
>   ): Promise<FileSystemDirectoryHandle>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/removeEntry) */
>   removeEntry(
>     name: string,
>     options?: FileSystemDirectoryHandleFileSystemRemoveOptions,
>   ): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemDirectoryHandle/resolve) */
>   resolve(possibleDescendant: FileSystemHandle): Promise<string[]>;
>   entries(): IterableIterator<FileSystemDirectoryHandleEntryType>;
>   keys(): IterableIterator<string>;
>   values(): IterableIterator<FileSystemHandle>;
>   forEach(
>     callback: (
>       param0: string,
>       param1: FileSystemHandle,
>       param2: FileSystemDirectoryHandle,
>     ) => void,
>     thisArg?: any,
>   ): void;
>   [Symbol.iterator](): IterableIterator<FileSystemDirectoryHandleEntryType>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream)
>  */
> export declare abstract class FileSystemWritableFileStream extends WritableStream {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/write) */
>   write(
>     data:
>       | Blob
>       | (ArrayBuffer | ArrayBufferView)
>       | string
>       | FileSystemWritableFileStreamWriteParams,
>   ): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/seek) */
>   seek(position: number): Promise<void>;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemWritableFileStream/truncate) */
>   truncate(size: number): Promise<void>;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle)
>  */
> export declare abstract class FileSystemSyncAccessHandle {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/read) */
>   read(
>     buffer: ArrayBuffer | ArrayBufferView,
>     options?: FileSystemSyncAccessHandleFileSystemReadWriteOptions,
>   ): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/write) */
>   write(
>     buffer: ArrayBuffer | ArrayBufferView,
>     options?: FileSystemSyncAccessHandleFileSystemReadWriteOptions,
>   ): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/truncate) */
>   truncate(newSize: number): void;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/getSize) */
>   getSize(): number;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/flush) */
>   flush(): void;
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/FileSystemSyncAccessHandle/close) */
>   close(): void;
> }
> /**
>  * Available only in secure contexts.
>  *
>  * [MDN Reference](https://developer.mozilla.org/docs/Web/API/StorageManager)
>  */
> export declare abstract class StorageManager {
>   /* [MDN Reference](https://developer.mozilla.org/docs/Web/API/StorageManager/getDirectory) */
>   getDirectory(): Promise<FileSystemDirectoryHandle>;
> }
> export interface FileSystemFileHandleFileSystemCreateWritableOptions {
>   keepExistingData: boolean;
> }
> export interface FileSystemDirectoryHandleFileSystemGetFileOptions {
>   create: boolean;
> }
> export interface FileSystemDirectoryHandleFileSystemGetDirectoryOptions {
>   create: boolean;
> }
> export interface FileSystemDirectoryHandleFileSystemRemoveOptions {
>   recursive: boolean;
> }
> export interface FileSystemSyncAccessHandleFileSystemReadWriteOptions {
>   at?: number;
> }
> export interface FileSystemWritableFileStreamWriteParams {
>   type: string;
>   size?: number;
>   position?: number;
>   data?: Blob | (ArrayBuffer | ArrayBufferView) | string;
> }
> export interface FileSystemDirectoryHandleEntryType {
>   key: string;
>   value: FileSystemHandle;

[jasnell]

This is already massive but doesn't yet cover everything. The various pieces may still be incomplete and buggy but the implementation is disabled by default without the additional pieces to expose the internal modules. Therefore, rather than let this continue to get bigger, let's get this part merged and I'll keep iterating in a different PR.

Note... there are likely api and implementation nits that need to be fixed. Those can but don't really need to be fixed in this PR still this is still all not exposed by default.

[anonrig]

I think this pull-request should land gradually, and possibly splitted into multiple pull-requests to review it in more detail. It's not easy to understand the data flow.

[jasnell]

I think this pull-request should land gradually, and possibly splitted into multiple pull-requests to review it in more detail. It's not easy to understand the data flow.

I can split it up but if anyone asks how bits and pieces fit together and why certain changes are needed in the individual small PRs I'm just going to point them back to this one.

[jasnell]

Moving this back to draft while I split out chunks into separate PRs