What's Next: Making a Kernel

[james-pre]

Hey everyone,

I wanted to share my plans and thoughts on what's next for ZenFS.

The Problem

At a high level, I'm really interested in adding features that bring ZenFS closer to native Linux. This includes unix sockets, procfs, sysfs, linear memory stuff like /dev/mem, and TTYs (which actually use sockets).

The primary problem with implementing all of these features is that they are fundamentally part of a kernel, not file systems. This problem has occurred with past features, and is the largest blocker of #222. Up until now, features have been implemented in a way that avoids implementing large portions of a kernel. Consequently, existing features are analogous to kernel features, but not fully featured. For example, procfs is based on processes. Linux processes exist as part of a kernel and are a lot more complex than the existing contexts feature.

The solution

I've thought a lot about this problem, and found a solution that I believe offers the most flexibility, modularity, and maintainability.

If you've read the ZenFS docs, you'll know that the current codebase is generally divided into two layers:

• The Virtual File System layer, which handles compatibility with Node.js APIs, mounting/routing, and permission checks
• The Internal* API, which handles abstracting backends into a single, unified FS interface and
  _{* In this context, internal refers to being conceptually "behind" the VFS— many parts of the "Internal API" are public and exposed.}

This has helped separate concerns and worked out well for making ZenFS modular. We can expand upon this design by presenting different frontends for ZenFS.

This diagram visualizes the idea well:

The Plan

Let's talk about the concrete steps that will be taken in the future:

• Breaking changes and major feature changes to the existing codebase will be paused
• The existing Node.js compatibility code will be split off from the Internal API. The Node.js compatibility layer will be the primary purpose of ZenFS and the @zenfs/core package in the future
• The kernel part of the codebase will be moved into a different package. Kerium already provides a solid foundation for POSIX code, so stuff will likely be moved there.

This will all take months, so any help would be awesome.

The Future

After migrating to the new architecture, there are some really cool things we can do:

• The aforementioned kernel features (e.g. /proc and /sys)
• Compatibility layers for new platforms, like C/C++ with WASM and maybe even Go or Rust (I only plan on implementing a C/C++ layer for now)
• New kinds of drivers that are not related to the FS, like graphics stuff.
• Potentially even complete emulators

The Discussion

I want this to be seamless and easy. If you have any thoughts on this, please drop a comment! I'm always happy to get feedback and would really appreciate it.

[Davilarek]

how would ABI function in that idea? would there be some real spec compliance in terms of those "drivers"? like said "web hid driver", would it follow some kind of common ruleset, thus allowing integration of existing software? there was a mention of C/C++, but would the project be limited to WASM layer or would it include some kind of processor emulation? this question is tied to my first question about the ABI. I like the idea behind syscalls usage here. what kind of information would be populated in /sys? procfs and sysfs are introduced by Linux kernel and not defined within POSIX spec. for a compatibility layer with C/C++, would that mean creation of an interpreter for the mentioned languages? or would that involve some kind of, again, emulation? the idea sounds interesting and complex but I'm not yet sure if there's a specific goal or if I'm missing something.

[james-pre]

how would ABI function in that idea?

The current plan involves primarily a syscall API. Any ABIs would be provided by frontends.

would there be some real spec compliance in terms of those "drivers"? like said "web hid driver", would it follow some kind of common ruleset, thus allowing integration of existing software?

As far as I know, there is no consistent singular API for interacting with JS devices. So, the API for drivers will need to be created independently. However, @mathiscode has already created numerous drivers using the existing devices API (which is really cool). It should be easy to migrate this to a new, more comprehensive driver API that isn't built specifically for emulating /dev.

There was a mention of C/C++, but would the project be limited to WASM layer or would it include some kind of processor emulation? this question is tied to my first question about the ABI.

ISA (instruction set) emulation will need to be implemented as different frontends. At the moment I only plan on personally implementing the Node.js frontend and a WASM frontend (that C/C++ can compile too)

I like the idea behind syscalls usage here. what kind of information would be populated in /sys? procfs and sysfs are introduced by Linux kernel and not defined within POSIX spec.

The goal is to (roughly) emulate Linux, since it's one of the few free open-source kernels that's widely used (1B+ devices)

Just like on Linux, /sys will be populated by drivers using a system similar to Linux's kobjects.

for a compatibility layer with C/C++, would that mean creation of an interpreter for the mentioned languages? or would that involve some kind of, again, emulation?

C/C++ support is relatively easy since they can be compiled to WebAssembly. WebAssembly can then be run, providing the syscall functions as globals, exposed via a small JS/TS compatibility layer (which does things like memory handling and paging).