Linux or our own kernel?

[el-bri-ga]

Should NineLives run on Linux, or should we write our own kernel? There are Rust crates now that do a lot of the heavy lifting, so writing one is less far-fetched than it used to be. We haven't decided, and I'd like to work it out here.

The case for Linux

Most of what's new in NineLives sits above the kernel: workloads that survive losing a machine, atomic upgrades, and system programs that keep their state on the Felix log. None of that needs a new kernel, and all of it is a lot of work on its own.

A kernel of our own would also have to cover what Linux already does:

• Drivers. NVMe, real NICs, whatever each cloud exposes. Most of Linux is drivers, and no crate covers that.
• Storage semantics. Felix's storage trusts fsync, pread, preallocation that doesn't change the file size, and optionally io_uring. Recovery is only correct because those behave exactly as documented. A new kernel and filesystem would have to earn that trust.
• Networking and async. Felix runs on tokio and quinn (QUIC over UDP), on top of std. A new kernel means a network stack and either a port of that runtime or a rewrite.
• Security and SMP. Isolation, getting fixes out quickly when something breaks, and a scheduler that holds up on many cores.

The case for our own kernel

• A much smaller trusted base for a node whose only job is running workloads off the log.
• Deterministic scheduling and I/O, which would make replay stronger than anything Linux can promise.
• Syscalls designed around the log instead of files and sockets.
• Booting, paging, interrupts and a basic scheduler are weeks of work with today's crates, not years.
• Nobody else is building an OS with a replicated log at the center, so there's a real research story.

Rust projects worth looking at

• Asterinas: a Rust kernel that aims to be compatible with Linux at the syscall level, so felixd could in principle run on it unchanged.
• Hermit: a Rust unikernel. A node that only runs Felix and felixd could be a single image.
• Redox: a full microkernel OS in Rust with its own userland.
• Theseus: a research OS built around live evolution and fault recovery.
• rust-osdev (bootloader, uefi, x86_64): the building blocks for writing our own.
• Rust for Linux: a middle path, with Rust modules inside Linux where we need something in the kernel.

Open questions

• Is the first goal a cluster that survives losing a machine, or a kernel designed around the log? The answer probably decides this.
• Would starting on Linux make it harder to move later, or can we keep felixd's dependence on the kernel narrow enough that it doesn't matter?
• Is Asterinas close enough to Linux that it's an experiment rather than a rewrite?
• Would single-purpose nodes (Hermit-style) be enough, or do we need a general-purpose OS on every machine?
• What would we lose in debugging, tooling and hardware support, and is it worth it?

If you have a view on any of these, or know of a project I've missed, reply here.

[el-bri-ga]

I read up on why Plan 9 didn't spread, since it's the closest thing to what we're trying to do. I'm putting what I found here as evidence for this question, not as a decision.

Migration. Pike's own summary (TUHS mailing list, January 2023) is that as a product Plan 9 failed for many reasons, including existing systems that were good enough, while as research it was a big success. Raymond makes the same point in The Art of Unix Programming (ch. 20): it wasn't enough of an improvement to make people leave Unix. The lesson I take is that whatever we pick has to run the software people already have from the first release.

APE's limits. Plan 9's answer to existing software was APE, its ANSI/POSIX environment. Trickey's APE paper is candid that it targets POSIX 1003.1-1990 and can't honestly emulate uid and gid changes, link, controlling terminals, fcntl locks, signal masks or umask. A compatibility layer that covers most of POSIX still left real programs out. That's worth keeping in mind for any kernel that claims Linux compatibility at the syscall level: how much of what Postgres or a JVM actually uses does it cover?

Drivers. Few drivers and few applications are the reasons usually given next to licensing. It's the same point as the drivers bullet in the case for Linux above, and the history backs it up.

Licensing. Plan 9 went through several restrictive licenses before becoming MIT in 2021, and the Plan 9 Foundation said the confusion had held back outside development. We're MIT already, so this one doesn't change the choice, except that a Linux image means honouring GPL source obligations, which is in the design doc.

What survived. The Plan 9 ideas that made it out (UTF-8, /proc, 9P inside Linux, QEMU and WSL) were the ones that could be adopted on their own, on top of an existing system.

My reading is that this all points towards an existing kernel, since most of what's new in NineLives sits above it. It doesn't settle whether a Linux-compatible Rust kernel like Asterinas is close enough to be an experiment rather than a rewrite, which is still the open part for me. I've written the general constraints into the design doc as "Lessons from Plan 9" (#27) and kept the kernel choice open.

Sources: Pike, TUHS, 2023-01-31; Raymond, The Art of Unix Programming, ch. 20; Trickey, "APE: The ANSI/POSIX Environment"; the Plan 9 Foundation's 2021 announcement.