Skip to content
Go to file
Cannot retrieve contributors at this time
300 lines (211 sloc) 11.3 KB

Alternate C libraries

Ebusl (pronounced like you think is would be: “E buse l”), is a hacked up version of the C library with five, maybe six, major modifications to it (planned, thus far).

The End of errno

The biggest change to musl is that errno is killed, dead. Everything makes an error return embeds the error return as part of the return. So, mmap for example, returns a negative value with the 10 possible errors encoded into it. There’s a side function that can turn that new value into the moral equivalent of errno and look it up, but that’s it.

There’s an increasingly misplaced errno related rant in the “traps” section of this document.

errnos are (too) stable

The number of error returns in POSIX is standard and stable and hasn’t changed in decades. But they are semantically separate (the return from the function indicates a simple error, the more complex error is encoded, sometimes badly, into an errno), and incomplete.

Why look up errno? why store it in a weird place (via TLS) that’s expensive to get to? Why do it in two phases? I really don’t know much about the history of errno, I think it is done the way it is because:

  • we had very narrow architectures with limited address space and registers
  • twos complement arithmetic had not swept the world
  • it seemed like a good idea at the time

The errnos are weakly defined and depend on context

And often are a poor map to what actually went wrong. Take sched_yield as one example. In a hard(er) real time system, it means “put me to the back of the run queue”. But that’s not what happens when called within regular Linux process, nor does the call actually return a value that said - yes - some other thread or process ran.

What you do with an EAGAIN errno varies depending on your underlying system architecture. EWOULDBLOCK (What it is currently mapped into), is semantically more clear - but the two return types are (currently) mapped to the same thing. An EWOULDBLOCKWHY message would often be helpful - “out of cpu?”, and in more than a few cases ENOSPC would be a better return than a mere EAGAIN.

Stuff like EINTR is usually easy - just retry immediately.

If you think about it, there are all sorts of weaknesses and oddities in many posix calls, that we’re just used to, that could be improved.

A few more examples - nearly everywhere a value of 0 means I didn’t do anything (write) OR “the call succeeded” - and a negative result is an error. Often, a 1 or greater is a handle to an opaque data type, and ghu help you if you pass that handle to the wrong function. There are a few interfaces (qsort comes to mind) where -1 is valid. Others that perhaps should return 0 - “I couldn’t open stderr because it’s mapped to null”, or “I didn’t write anything just now” don’t, they consider it an error.

Asynchronous operations are a big problem, also - I can allocate 8GBs but when I run out of swap, bad things happen.

Same goes for atomicity.

errno was designed to work with down to 8 bit systems

Where today we are moving to 64 bits and beyond.

Kill the errno

It looks like I have to re-org wrappers around something like 80 linux syscalls to do things this way, and it’s kind of my hope to use a modern tool (like coccinelle) to manage the patchset and for that matter automate the process.

‘cause everything that calls these (and there are only few things planned so far - tcpdump, libpcap being the biggies) also has to have all references to errno ripped out.

Some of this is easier - a lot of stuff just calls perror without referring to the errno at all, and in this case “eerror” would just be passed the return from the function.

Some of this is hard: a lot of code checks for -1, where here we check for “-“. A lot of times we don’t need to “check” at all - on all modern 64 bit arches an invalid address causes an invalid address trap.

Since we encoded the actual error into the address, it can jump to the right thing, also.

Other stuff may just inline an invalid instruction (which gets trapped) - or call trap directly with an offset.

I’ve thought about killing NULL this way, also, but try to restrain myself. Watch out! I STILL MAY! It is more likely, that I’ll try to fold in NIL, and NAR (not a result) - as a more comprehensive four valued logic.

I totally realize that this change makes erm code incompatible with everything else, but there are already other things like a separate call and data stack that do that, too.

There are only 277 direct references to errno in musl.

A better future for errnos?

At the very least, seeing the range of errnos start to evolve again in mainline posix to better represent new realities would be a goodness, I think. Instead of arguing about which posix errno to return in the case of ambiguity, we’d be trying to resolve the ambiguity with a new error definition in posix.

Bugs accumulate in the gaps between interfaces, and work on improving the rigor and expressiveness of those interface standards in face of changing reality is increasingly needed.

Weirder threads

  • In POSIX: Threads are pre-emptive, and global, and carry around their own state via tls.
  • In ERM: Threads are co-operative, and per cpu, and keep their own state in their own declared areas.

To do this, in part, involves killing the biggest user of TLS, dead - which is errno. And god knows what else.

I HATE errno. It’s good for humans, lousy for encoding actual errors.

Better support for traps

ERM tends to trap on an error - and not actually do any inline error checking. What you do instead is: declare a trapped function to the compiler, and on error, you raise a trap. You overrun memory - you trap. You fail an operation - you trap.

C++ has exceptions. These aren’t it.

GO went so far as to include a second variable entirely from all returns. You then need to handle those inline. Yuck.

C has errnos. Show me one program that actually checks for and does something correct for every possible errno a function can return, and I’ll buy you lunch. Maybe the space shuttle did, I don’t know. We’ve certainly crashed enough other spacecraft.

LISP has hooks you can put in front or behind anything. Hooks are really nice. A trap in erm looks a lot like a hook - except that they aren’t inline. The linker has a map against the address of that bit in the code that it builds a set of error handling hooks into.

“I got an instruction trap from PC counter of X - what handler do I call?”

Most ERM traps are statically declared. They MUST handle all possible error returns from the function. You can certainly create a set of traps and reuse those conditionally, inheret, copy/paste, whatever, and hopefully most of the time, any given program won’t need more than a few custom ones.

There is one huge advantage in using traps. You can easily find and simulate everything that can go wrong in the program and how it is handled. You can more prove it is correct.

Another advantage (and this is not always feasible) - is that your code doesn’t get cluttered with inline error handling.

A disadvantage is that traps are slower than branches (unless I really work hard to speed them up). And it’s not always obvious what your code will do as the trap handler is defined elsewhere from where you are handling the error.

I’m not really sure to what extent traps will make it into ebusl. It’s a really big job to just kill errno.

Separate call and data stacks

This makes ERM more resistant to ROP, in particular.

There already isn’t much of a data stack (the hope is that nearly 100% of the time args are passed in registers, and most other communication is via message passing), and while the call stack doesn’t expressly forbid recursion, nothing more than tail recursion is encouraged.

Message passing

Message passing, as defined in the 80s and 90s, was easy due to a paucity of registers. It is hard to use up the modern plethora of registers if all your functions do is send(somewhere, data);

ERM has a lot of intentionally disjoint memory spaces, where message passing is the only way to get data in or out, and DMA is used on bigger stuff rather than involving the main cpu.

Structure passing

The C compiler (due to C++) has got a lot better at packing structures into registers.

Structure return

It’s considerably less good at packing structures into registers on a return.

Bitwide types

It rather bugs me that bitwide types cannot (still) be easily packed together. Ideally C would have developed a bitsX type by now instead of always promoting things to a “natural” quantity, and you’d be able to arbitrarily declare (and pass) a complex variable with a:2, b:4, c:5 without having to resort to #defines. Specialized versions of C (and C++) exist (specc, systemC) that can do this, and more than a few times I’ve been tempted to just start writing in those…

… but unwilling to give up so much performance and compatability for the sake of simulation.


I don’t know what to do about printf and varargs in a message passing system, as yet. I certainly would like to have a drop in message passing replacement for printf.

As written, however, there is no floating point in erm, at all, and that’s a whole lot of code that can be compiled out.

Someday FP might land back in it, though.

And printf is really well defined, and bloody useful.

Shared memory timers

The only thing that really bugs me about musl itself is that they didn’t copy glibc’s WONDERFUL shared kernel memory timer implementation. I would like to do that (for mainline musl as well) as it shaves MANY nanoseconds off of getting timers - like orders of magnitude - and doesn’t trap to the kernel at all.

Other problems

I have no idea what else will break. I’m scared to look.

Alternate LIBCs


waaaaaaay too big and ancient.


undermaintained. It WAS what I hacked up last time, but getting it to work right with C++ was a PITA. Last time I was also trying to get away from needing virtual memory, also, and I’m not sure to what extent musl works without virtual mem.


has quite a few compelling advantages - it’s small. It’s used a lot. It’s the default lib on the parallella.

but it is not anywhere near as feature-full or posix compliant as musl.


I can say, without cynicism, that Linux is the greatest C library ever invented. There are so many things that are easier to do within the linux kernel than in userspace, that the concept of folding in concepts from erm into into the kernel instead always remains tempting.

However - although I keep thinking I’ll utilize librcu or something like it at some point, getting at advanced processor features (like the SSE registers) tends towards being suited only for bulky, higher latency operations there. Other forms of hardware assistance are also hard to do.

Other libcs

I haven’t looked at bionic.

Calling convention problems

It seems highly likely I’ll have to muck with the default C calling conventions for the various architectures.

A lot of state does get encoded into a few static registers already. And structure return is difficult, yet important.

I’m not looking forward to this - because then I’ll have to mod the compiler, too. I’m already planning on abusing C in lisp-y ways, I am tempted very much to already start using a set of UTF-8 characters everywhere, like assignment:

←(assignedvalue, operation(x,y,z));


The plan is to work on that crazy part of the project in a separate repo, using git submodules to bring it in.

You can’t perform that action at this time.