How to deal with breaking changes on platform ? [BSDs related]

[semarie]

I open an issue on libc because it is here the problems will start to show up. Depending the solution or the way to deal with, modifications could occurs in rustc repository too.

At OpenBSD, we don't care about breaking API/ABI between releases. Once a release is done, the API/ABI is stable, but there is no guarantee that it will be compatible with the next release.

Currently, in the upcoming 6.2 version of OpenBSD (6.1-current), there is a breaking change that will affect libc : si_addr should be of type void *, not char * (caddr_t). Here the current definition in libc.

Under OpenBSD, we deal with ABI under LLVM by using a triple like: amd64-unknown-openbsd6.1. For Rust, instead we use an unversioned platform, resulting all OpenBSD versions to define the same ABI (which isn't properly right).

Do you think it possible to switch from *-unknown-openbsd to *-unknown-openbsd6.0, *-unknown-openbsd6.1, ... without having to duplicate all code in libc for each target ? and without having to add a new target in rustc for each OpenBSD release ?

Any others ideas on the way to deal with it ?

[alexcrichton]

Unfortunately I don't really know how we'd handle this, I just figured that platforms wouldn't do this.

If this happens a lot we'll just need to document what's wrong and stop adding new bindings, it'll be up to crates to implement version compatibility.

[semarie]

I think it isn't just a "version compatibility" issue. My purpose isn't to have a compatibility layer for missing/removed functions or types.

The problem is OpenBSD triple is versioned, meaning that API/ABI of one version could be different from another version.

I checked some others system (running llvm-config --host-target to see if the triple is versioned or not), and it seems it is a common situation in not-Linux world:

• x86_64-apple-darwin16.0.0
• x86_64-unknown-freebsd12.0
• x86_64-unknown-freebsd11.0
• i386-unknown-openbsd5.8
• x86_64-unknown-netbsd7.99

I also checked in LLVM source tree: the OS version is a part of the triple definition.
see getOSVersion() in include/llvm/ADT/Triple.h.

Maybe a concept is missing in Rust ? If target_os_version would be available, it would solve the issue: parts that are only defined in some OS version could be isolated from another OS version.

I don't think it is a problem only on OpenBSD. Any OS using OS-Version could be hitted. OpenBSD exposes it because we heavy use the ability to not be API/ABI compatible (it is a way to be able remove old stuff that deserve security).

[alexcrichton]

Yeah there's no concept of a versioned target in rustc right now, and we're unfortunately not really capable of doing so right now.

Our only recourse is basically to take the subset which currently works across all revisions, put that in libc, and then otherwise let downstream crates bind versions that change over time.

[semarie]

I hope you are kidding: you are asking to remove siginfo_t type for OpenBSD from libc and so to break stack_overflow detection for OpenBSD (libstd relies on it). And even if we can drop stack_overflow detection, it doesn't resolv the intrinsic problem.

So I am looking to extend Target to include os-version information in the target specification.

[alexcrichton]

Well, I'm not really kidding. If we feel we must fix this then we currently have no choice but to not add the bindings. If we don't want to do that then the fix must go elsewhere. I don't know the best way to fix this, just spitballing.

[asomers]

This isn't just a problem for OpenBSD. FreeBSD 12, when it comes out, will change a number of important types, like ino_t and struct stat. If libc's policy is to only bind the greatest common denominator between versions, then overtime it will shrink into irrelevance. Such a policy really just kicks the version compatibility can down the road.

Would it be possible to generate bindings dynamically at build time? When writing Ruby bindings, I've always preferred that approach to FFI. If not, then I think libc needs a way to distinguish between OS versions, just as it currently distinguishes between OSes.

[alexcrichton]

I would personally be afraid of generating bindings at compile time. It just pushes the problem to consumers without giving them tools to deal with it.

I do think that this sounds like this needs a way for libc to distinguish between OS versions, but Rust currently has no tool for doing so really.

[asomers]

The problem just got worse. Linux 4.11, released today, added a new system call: statx. Until libc learns to understand versions, it can't add support for statx. I really think that cargo needs some sort of configure step analagous to autoconf's configure.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.11-Statx-System-Call

[alexcrichton]

@asomers that's not quite true though, we can add bindings at any time. Rust supports Linux 2.6.18+ and there are a huge number of syscalls bound in libc not present in 2.6.18. It's up to crate authors to pick and choose apis for platform compatibility appropriately.

[asomers]

@alexcrichton I guess I was wrong about how libc's CI tests worked. Are you saying that libc's tests do not flag symbols defined in FFI but not present in the system's headers? If that's true, then a Rust program trying to use statx on Linux <= 4.10 would build but get ENOSYS at runtime, right? That's better than what happens if a Rust program tries to use aio_waitcomplete on FreeBSD 10, where the FFI binding would actually be wrong. But Linux is not immune from changing syscalls, either. The first example I could find was utimensat. Its signature changed in 2010, well after 2.6.18 was released. Any Rust program using libc will try to use the new version of utimensat, even when built for older systems.

Would you consider dynamically generating the bindings for select functions, even if most functions have static bindings? Right now, I don't see any way at all for consumers to deal with the versioning problem.

[alexcrichton]

No, to be clear:

• The libc crate is basically just a header file.
• The libc crate is automatically verified on many platforms, but we have exceptions in libc-test/build.rs. It's not guaranteed that the verification passes on every instantiation of a platform.
• If you use a function from libc you're referencing a symbol.
• If that symbol doesn't actually exist on your system, you'll get a linker error.

Programs using statx will likely get a linker error and will then have to deal with that appropriately.

I would like to avoid dynamically generating the bindings, as that's typically not the actual solution to this problem. It makes cross compilation (even just across OS versions) much more difficult

[asomers]

Cross-compilation could be solved by overriding the build script's platform detection. For example, on FreeBSD there's basically only one symbol that a build script would need to detect: __FreeBSD_version. When cross-compiling, cargo could set that in an environment variable and the build script wouldn't try to detect it from the system headers.

But it sounds like you might have something else in mind when you say it's "not the actual solution to this problem". Do you? What is the "actual solution", @alexcrichton ?

[alexcrichton]

To me the "actual solution" is precisely what we're doing right now. We list a bunch of symbols and authors need to be vigilant about which ones they use. This does not solve the use case of OpenBSD, however, if there are ABI breaking changes. We may need more than one solution but to me there are too many downsides to dynamically generating an API on Linux at least.

[asomers]

Not only does it not solve OpenBSD's use case; it doesn't solve the use case where operating systems make changes that don't break the ABI. Both FreeBSD and Linux occasionally change syscalls and provide backwards compatible syscalls with the old signature and syscall number but a new name. For example, FreeBSD 8's "compat7.shmctl" syscall is identical to FreeBSD 7's "shmctl". Similarly, operating systems make changes to system libraries and provide backwards compatibility by bumping the SHLIB version and providing the old libraries as optional packages.

Currently libc handles neither of these cases. Either the libc binding tracks the new function's signature, which breaks Rust programs at runtime on older versions, or the libc binding stays with the old signature, which breaks Rust programs at runtime on newer versions. It's simply not possible for the current libc to compile correctly on multiple versions of an operating system. Your previously suggested solution is to simply remove a binding whenever the OS changes it. But that would break any crates that use the binding, violate semver, and still result in runtime failures for crates that use the old libc but were built on a new OS.

You suggest that libc's consumers should be responsible for versioning issues, but I don't think that's possible. Let's take stat(2), for example, which will likely change in FreeBSD 12. Suppose that when FreeBSD 12 is released, somebody tries to compile the nix crate on it. The linker will be satisfied that stat is present in libc, but the signature will be totally wrong, so nix will fail at runtime. Cargo won't produce any kind of warning. If I understand you correctly, you suggest that stat should at this point be removed from libc. But that won't fix nix until somebody updates that crate's dependencies, and even then it will only change a runtime failure into a compile time failure. Must the nix developer then write a build script that checks __FreeBSD_version and reimplement all of stat's FFI bindings for FreeBSD 12? That would finally fix the problem. But according to crates.io, libc has 1268 dependent crates, and all of them would have to independently write the same build script and add the same FFI bindings for stat on FreeBSD 12.

Alternatively, libc could assume that all operating systems provide backwards but not forwards compatibility (sorry OpenBSD). Then it could pick a minimum supported version, and always link against that version's shared libraries, Currently Cargo doesn't provide a mechanism to specify an exact shared library version to link against, but that could be added. This would fix all of the runtime failures, but at substantial cost: dependent crates would lack access to new OS features, and both developers and users would have to install the compat library packages. Not only would new features that change APIs be unavailable, but the shared library lock would mean that entirely new functions would be unavailable as well, unlike the current situation where using newly added functions will generate link failures when building on an old OS.

In either case, developers will likely fork libc to update their favorite bindings, resulting in a Balkanization of libc and dependent crates that don't support older OS versions.

I understand that cross-compilation is a really cool feature, but I fear that you're underestimating the severity of this problem. Have you looked into how embedded cross development toolchains work? AFAIK the host system requires full headers for the target. Maybe Rust needs to do the same.

[alexcrichton]

@asomers if you've got a proposal of what to do I'd recommend writing up an RFC, with so many dependencies changes such as what I think you're proposing can't be taken lightly.

[semarie]

@alexcrichton I pushed a WIP branch on my github repository. I hope code will be more explicity than my explaination about what I called having support for OS version.

Tree is at https://github.com/semarie/rust/tree/target-os-version . Please note my code isn't working for now.

Basically, it is:

• extending Target to embedding a (possibility empty) os-version string
• exposing the string as target_os_version symbol (in the same way than target_os)

It would be possible to do have conditionnal code against OS version (OpenBSD 6.1 or OpenBSD 6.0), in the same way we have conditionnal code against OS name (OpenBSD or FreeBSD).

[asomers]

Good work @semarie. BTW, I've been studying ELF symbol versioning and I think it would be possible to fix libc without modifying Rust itself. Basically, libc would need to grow a bunch of feature flags like "freebsd11+", "freebsd10+", etc, meaning "build code that will work on FreeBSD 11 or greater" and "build code that will work on FreeBSD 10 or greater". Of course, those flags could be conditionalized so they won't appear on other platforms. Then, for every symbol that differs between FreeBSD versions, libc will bind a different version depending on which feature flags are set. The link_name attribute will encode the specific ELF symbol version number used on the oldest OS version chosen. I don't have code yet, but I think this approach will work for FreeBSD and Linux. Does OpenBSD use ELF binaries or is it still using a.out?

Also, I've found several functions in glibc with multiple versions. Linux is not immune from this problem.

[semarie]

@asomers OpenBSD uses ELF on all platforms. but using ELF symbol versioning doesn't help for breaking changes if the OS doesn't use symbol versioning.

[alexcrichton]

@semarie I'd personally probably reocmmend writing an RFC before sending that as a PR, I'm sure many others would have comments as well!

[raphaelcohn]

This has hit me too - in particular, with changes in Mac OS X major versions. However, a good solution probably isn't to use a version number in the target triple, as there's a distinction to be drawn between libc version and OS version; they do not necessarily go in lockstep. A classic example might be changes to Linux's uapi headers, which don't yet line up with changes in musl, say.

This problem is a general one: changes in third party (usually C) library APIs that are incompatible. It needs a good solution within Rust. It's a problem that's heavily compounded by set ups that use dynamic libraries (something I've come to see as more trouble than they're worth for secure or robust systems outside of the desktop. In practice, it's a far too difficult for most sysadmins to assess whether a security fix to a dynamic library affects more than on running program, and so they just go for the nuclear option of a reboot). Using autoconf like tests or dynamic bindings at runtime is probably the wrong way to solve this generally. Such approaches require too much of the system they are on (execute permissions, existence of compilation-associated tools, headers, etc). They are also deeply unfriendly to security audits and locked down systems (eg those built entirely from source). autoconf in particular makes the classic mistake that the build system is similar to deployment; it's always been an absolute beast to get things to cross-compile with it repeatedly, robustly and consistently. Too many things (eg time-of-date, location of shell interpreter, absolute sysroot paths, etc) creep into the deployed solution.

(Semantic version does absolutely nothing to solve this; in fact, semantic versioning is a deeply broken concept that's become popular recently. In practice, either a version is compatible or it isn't; semantic versioning is just the upstream's author's assessment. One man's inconsequential security version or minor change is another's nightmare. In practice, with large system set ups and deployments, I always encourage dev teams to think of only two kinds of version: likely-to-be-compatible security fix, and incompatible. Everything incompatible needs to go through the full test cycle before deployment. Security fixes can bypass that if urgent; risk vs reward and all that).

[comex]

Er, is the siginfo.h change in question actually ABI-breaking? char * and void * should have the same memory representation, so I'd expect that change to break the API (for newly compiled C code) but not the ABI.

Though it seems there's a more general problem to be solved here.

[semarie]

@comex yes, the change char * to void * is just an API break regarding OpenBSD. But it is an uncommitable change in crate libc without major version bump.

I started a discussion on internals, and I am working to submit a RFC.

[Rufflewind]

Related: rust-lang/rust#42681

fs::metadata() crashes on FreeBSD 12 due to layout change in stat.h

[mattmacy]

#775

I don't quite understand how Rust initially missed out on OS and ABI versioning quite so badly - deciding to assuming that structures and types are immutable over time or removing key structures from libc altogether. Nonetheless, Rust can conditionally compile based on configuration values, what is stopping this?

[semarie]

@mattmacy my understanding of the problem is:

• build time configuration adds more complexity for crosscompiling (you need to target a particular OS ABI)
• crosscompiling is used a lot in Rust infrastructure (for testing, for rustup...) So using OS ABI would mean a potentially high number of new targets to check, and binaries to produce, resulting an infrastructure more complex to maintain (it needs to scale)
• it is only a problem for BSDs, and they are not a high priority

[semarie]

I am still sad about that.

Please remember that currently FreeBSD 12 uses a compatibility layer for ino_t (FreeBSD 11 uses 32bits version, and FreeBSD 12 64bits) and silenciously truncate inode values if not representable on 32 bits. One day it will hurt, and it could be badly.

Refusing to accept the reality that OS introduces breaking changes from time to time, and enforcing that at libc level is bad.

[gnzlbg]

Refusing to accept the reality that OS introduces breaking changes from time to time, and enforcing that at libc level is bad.

That OS introduce breaking changes is a fact, and AFAICT everybody agrees that this is a problem.

I personally haven't heard a solution to this problem that did satisfy our constraints and did not have major downsides. Most proposed solutions have a high cost and the actual added value of solving this problem is unclear - this is a very tricky problem to solve correctly.

For the FreeBSD case, the ""best"" solution might be to just bump the minimum supported FreeBSD version to FreeBSD 12, and if there is enough demand, to add a ...-freebsd11 target, or to provide an easy way to compile to such a target using xargo. But all of these changes are breaking, so they aren't great either =/

[asomers]

For FreeBSD, I think the easiest solution would be to add a default-on freebsd11-compat feature flag to libc. When on, libc binds the FreeBSD 11 version of symbols like struct stat. When off, it binds to the FreeBSD 12 version. This wouldn't require introducing any new target triples. However, that technique wouldn't help OpenBSD users. OpenBSD pretty much needs a new target triple for each version, since it doesn't preserve any backwards compatibility.

[raphaelcohn]

@gnzlbg @asomers Whilst such techiques are likely to work, the downside is that a large number of 'legacy' triples accumulates over several years. Over a decade, a large number of the *BSD triples would effectively be obsolete, and hence challenging to maintain (The "matrix of pain" grows too large even for an experienced team).

One only has to look at GNU's disastrous config.sub / config.guess code to see what can happen. (They also show why target triples are exceedingly naive; at least Rust has gone down a much cleaner route by defining toolchain features with its JSON target files and dumped the idiocy that is feature testing).

The challenge for what to do is harder still when one uses C libraries that are NOT tied to a particular OS release, yet occasionally change to maintain compatible. A classic example is musl, which adds features with each release to improve its 'compatibility' with POSIX and other C libs, and every now and then also changes to accommodate a Linux kernel revision. (I've argued previously for separating core C library features from Linux headers, but the two are so tightly coupled in some views of the Linux world that it's impossible).

A better long-term approach may be to actually slim down libc to just a sub-set of POSIX, and then provide companion crates, eg for musl, glibc, ulibc, FreeBSD X.X, etc, which contain library-specific features. It may be that a companion crate might actually cover several very closely related libraries (eg musl + glibc + ulibc), but even that has difficulties. And even if there's a package per c-lib, there still needs to be a sensible way to handle c-lib versions that are moderately recent. I suppose eventually one just ends up with a lot of feature flags.

Of course, this gets challenging when also needing to consider Windows - but the vast majority of useful Windows functionality is outside of the library in any event.

[gnzlbg]

@raphaelcohn I agree that these approaches have downsides. I agree with you that:

A better long-term approach may be to actually slim down libc to just a sub-set of POSIX, and then provide companion crates, eg for musl, glibc, ulibc, FreeBSD X.X, etc, which contain library-specific features. It may be that a companion crate might actually cover several very closely related libraries (eg musl + glibc + ulibc),

And for windows the winapi crate already does this. For macos there are actually a couple of system libraries, and these are not exposed in one but in many crates. So each platform could have its own crate (e.g. freebsd, openbsd, etc.) with its own release schedule, and as long as the fundamental types (e.g. c_int) are defined somewhere else, all of these can interoperate.

The downside of this approach is that "slim down libc" is a backwards incompatible change. We might be able to extract the parts of libc that libstd actually uses into its own crate, and make sure that libstd is as "version independent" as possible.

[ssokolow]

A better long-term approach may be to actually slim down libc to just a sub-set of POSIX, and then provide companion crates, eg for musl, glibc, ulibc, FreeBSD X.X, etc, which contain library-specific features. It may be that a companion crate might actually cover several very closely related libraries (eg musl + glibc + ulibc), but even that has difficulties. And even if there's a package per c-lib, there still needs to be a sensible way to handle c-lib versions that are moderately recent. I suppose eventually one just ends up with a lot of feature flags.

What a coincidence. I don't see much on this topic for years and years and now, in the same month, not only you but also the Linux developers were discussing ideas in this vein:

• https://www.phoronix.com/scan.php?page=news_item&px=Linux-Propose-System-Wrap-Lib
• https://lkml.org/lkml/2018/11/10/470

[raphaelcohn]

@gnzblg I agree it would be a backwards incompatible change; it really depends how it is handled. At the moment, libc is 'stuck' in a 0.2 series of releases and so there is an opportunity - although it would require a major effort. Something like 6 months full time, I suspect, just to get a new, robust and agreed shape together.

I was thinking a little more over the weekend inspired by

with its own release schedule
and I think that would also allow a solution to version legacy and its associated maintenance. A new major release of a per-platform-like library would be free to drop support for an older version. Alignment with OS releases (or at least, with deprecation of long term support releases and the like) becomes quite possible, but historic code, because of cargo's excellent versioning, could continue to work.

Such a splitting out would also make it easier to introduce pull requests for very platform specific features, and would lessen the burden on those maintaining libc to know all of its oddities on many platforms.

@ssokolow Interesting. However, like communism, these sorts of centralizing approaches always seem to fail in practice (unless there's serious money to be made). It's a bit like the sirens; the call is alluring and impossible to counter, yet the reality is often ugly.

I was pondering on how best to tackle a newer libc structure, too, to make it easier to maintain and observe where things are 'missing'. Something that seems useful is to have a layout of code in files which mirrors the public headers. Perhaps headers could even map roughly to modules? There can't be a 1:1 mapping, and there would always be exceptions, but it might make some things a touch easier.

[gnzlbg]

FYI the llvm-sys crate provides bindings to different incompatible LLVM version, and it does so by maintaining different branches for each LLVM version, and different major crate versions: e.g. llvm-sys versions 50.0.0, 60.0.0, 60.1.0, .. which correspond to LLVM versions 5.0.0, 6.0.0, 6.0.1, etc. The patch version number is used to release newer versions of the crate.

That is, users just include the bindings for the LLVM version that they want to target by setting e.g. llvm-sys = 60 and then they get the latest bindings for that version.

If somebody wants to develop a crate to target, e.g., different versions of some OS, that's an strategy that might be worth considering.

[ssokolow]

@raphaelcohn

However, like communism, these sorts of centralizing approaches always seem to fail in practice (unless there's serious money to be made). It's a bit like the sirens; the call is alluring and impossible to counter, yet the reality is often ugly.

I have no idea what you're talking about. What centralizing?

The idea proposed is that, since glibc sometimes lags behind on wrapping Linux syscalls, maybe the Linux kernel devs should provide a Linux kernel syscalls library to ensure that they're available in a timely manner. It'd be no more centralized than having the same people maintain both the kernel source and the kernel menuconfig to make sure they stay in sync, which obviously works out just fine.

[raphaelcohn]

@ssokolow Given the way you describe the Linux kernel syscalls library proposed, then I don't see anything about it that couldn't also be provided by a similar Rust wrapper. That said, a large part of the effort in libc's like musl is cleaning up the inconsistency in Linux's syscalls and associated data structures and occasional mismatch with POSIX APIs.

[inferiorhumanorgans]

Would it be possible to back up a bit and determine the ideal support matrix? For example, should libc limit support to the actively supported versions of an operating system (e.g. FreeBSD 11 & 12, OpenBSD 6.3 & 6.5, DragonFlyBSD 5.4)? If so it seems like while there would be some churn in the actual flags, the maintenance burden shouldn't be so high. With such a narrow support matrix the following config flags could, in theory, work: unix, bsd, freebsd (a.k.a. freebsd12), freebsd11, freebsd12.

Regardless, IMO, the ABI ought to be indicated somehow regardless of whether the ABI is tied to the OS release (e.g. OpenBSD) or the library itself (e.g. glibc 1 vs 2).

In either case the top-level README seems a bit out of date in terms of what's actually being tested.

[asomers]

So far people have proposed 4 basic solutions to this problem, and they're all inadequate:

• Adding a build script would break cross-compiling.
• Changing target triples into target quadruples (eg amd64-unknown-openbsd6.1) would be way too much work.
• Adding backwards-compat feature flags (eg freebsd-11-compat) doesn't work for non-backwards compatible changes, such as OpenBSD's.
• Splitting libc into subcrates and keeping only the minimum functionality in libc itself isn't really a solution at all; it just kicks the can down the road.

In addition, none of those solutions even attempt to address the problem faced by rare or proprietary OS forks. Right now if somebody makes a new OS fork, like Bitrig, TrueOS, or yet another shiny Android fork, the only way to get Rust support is to add a new target_os in Rust . That's way too much work for forks that will only change a handful of syscalls. Worse, adding a new target_os isn't even possible for proprietary forks like Isilon's OneOS. The only way to solve the problem is to absolve rust-lang/libc from the responsibility of maintaining C library bindings for every single OS version in the world. OS's need the power to vendor their own libc crate. I propose a new way to do this, which would require a small change to Cargo but no changes to libc. Since it's not a libc change per se, let's discuss it on the forum here: https://internals.rust-lang.org/t/pre-rfc-global-source-replacement-in-cargo-for-os-bindings/9383 .

[myfreeweb]

Adding a build script would break cross-compiling

We could generate bindings on the fly when natively compiling, and use stored bindings (the ones that exist now) when cross-compiling maybe?

[gnzlbg]

We could generate bindings on the fly when natively compiling, and use stored bindings (the ones that exist now) when cross-compiling maybe?

I think we can do this for consts, but if the "stored" bindings and the "native" bindings differ in type signatures, crates that compile against one might fail to compile against the other (because they use the wrong types, etc.). That is, crates targeting libc would need to be able to handle the "stored" vs "native" binding differences to be able to compile against both using cfg()s, and at that point we are back to "we need some sort of binding versioning and some way to cfg on that".

[raphaelcohn]

@asomers A mad, mad option 5: Build a libc in Rust, and use that as the default. It actually solves a very large amount of the cross-compiling problem, too. It's certainly do-able for Linux and the BSDs.

I've played around with all of the 4 basic solutions you've mentioned, and all are far from ideal. Once one adds in the various horribleness of different CPU architectures, it just gets plain crazy (eg PowerPC and MIPS on Linux) and maintenance is tough.

I do think one thing we should give up on is trying to have lots of #[cfg] options on structs and the like just so we define things in one place. I've tried doing similar things in my own code (eg wrapping some obscure socket options for Linux) and it's actually just too hard to maintain or understand. Instead, it's one of the few times in my career I'd actually advocate duplication. Each libc, for each OS, gets it own set of files, one per named header in the original libc if possible. I'd actually go further, and name the files by function name / struct and, if necessary, architecture, too, eg:-

• musl/sys/utsname.utsname.rs
• musl/sys/utsname.uname.rs
• musl/sys/signal.SIGWAIT.mips.rs
• musl/sys/signal.SIGWAIT.x86_64.rs
• musl/sys/signal.SIGEMT.mips.rs

Many will find this too fine a break down. I've come to find structuring definitions like this extremely useful. At a glance I can see that there isn't a definition of SIGEMT for x86-64. It allows for changes to be fine-grained, and easy to add files for a new architecture. And too show when a particular architecture or variant has an extra struct, flag or the like.

Personally, I'd be tempted to go further, and define:-

• musl/1.24/sys/utsname.utsname.rs
• musl/1.24/sys/utsname.uname.rs
• musl/1.24/sys/signal.SIGWAIT.mips.rs
• musl/1.24/sys/signal.SIGWAIT.x86_64.rs
• musl/1.24/sys/signal.SIGEMT.mips.rs.
• openbsd/6.1/sys/...

And then use the crate's default features to pull in the latest version when not otherwise specified.

[gnzlbg]

I do think one thing we should give up on is trying to have lots of #[cfg] options on structs and the like just so we define things in one place. [...] Each libc, for each OS, gets it own set of files,

This is how libc is already implemented ?

Personally, I'd be tempted to go further, and define:-

Breaking changes across OS versions are small, duplicating all the code for each version seems overkill.

A mad, mad option 5: Build a libc in Rust,

Go for it! You are not the first one to have this idea, and others have already started doing that. Maybe you can help them!

[raphaelcohn]

@gnzlbg Not in the slightest! It's an illogical mess attempting some deduplication with inconsistent naming. Not really anyone's fault - it's the result of Rust's successful growth - and it demonstrated just how hard it is to create a consistent-ish solution. (It's actually quite a common (and healthy) problem in open source projects with lots of contributors). I find it very hard to navigate and understand when checking for support of particular function or struct.

One does not really need to duplicate if one wishes - one can always use a symlink. Indeed, that is potentially actually quite a good option, as it logically shows that, say the struct foobar is the same in version 1.24 and 1.25. However, my argument is that this problem is one of the very few - indeed almost only - time in my 20 year career I'd argue against de-duplication and DRY.

[gnzlbg]

Not in the slightest! It's an illogical mess attempting some deduplication with inconsistent naming.

PRs welcome I guess. The amount of work required to clean this up (independently of how messy this currently is) is insignificant compared to the amount of work required to re-implement the libc's of all OSes that Rust supports in Rust.

I find it very hard to navigate and understand when checking for support of particular function or struct.

There are docs for all architectures that rust supports online: https://rust-lang.github.io/libc/#platform-specific-documentation You can click on an arch, click on a struct, click on [src], and every time I've used it, it brings me to the relevant source code.

[mati865]

@radupopescu you might want to check relibc.

[asomers]

@raphaelcohn please don't rewrite every OS"s C library in Rust. That's basically what Go tried, and it hasn't gone smoothly. It turns out there's some hard stuff in the C libraries. Plus, it doesn't help libc's portability problem. It just kicks the can down the road. As I described in that forum post, I think the solution is more decentralization; libc wouldn't have to support every single version of every single OS if Cargo allowed OSes to provide their own libcs.

[gnzlbg]

libc wouldn't have to support every single version of every single OS if Cargo allowed OSes to provide their own libcs.

I suppose this could work for "less severe" breaking changes, like changing the value of a const. But some FreeBSD breakage has been caused by, e.g., changing the type of a struct field.

If the OS replaces libc with its own one containing the fix, then what? AFAICT libstd, and all other code using that API, won't compile because of a type mismatch, right?

[asomers]

Actually, changing the value of a const is impossible to do in a backwards-compatible way. The FreeBSD breakage you mention is less severe, because it's all backwards compatible thanks to ELF symbol versioning. The problem arises because Rust ignores the system headers, which is necessary for cross-compiling.

If FreeBSD 12 were to install its own forked libc as I suggested, then libstd and rustc would still work just fine. The rustc built from ports would use the correct libc for FreeBSD 12. And the rustc built from rustup would use the correct libc for FreeBSD 11, which would still link and run just fine, but would lack support for FreeBSD 12 features like longer mountpoints and 64-bit inodes. Problems would only arise if you assume that, for example, libc::stat is identical to libstd::stat. I doubt there's much if any code that does that.

[gnzlbg]

Actually, changing the value of a const is impossible to do in a backwards-compatible way.

I meant the Rust const, that would be, e.g., a C macro (#define FOO 3), an enum discriminant, etc. Those do not appear in the ABI of the C library (no symbols), but they need to be exposed in Rust. Replacing libc with a different one should be able to update the value of these, but I don't think it can update their name without causing breakage on code that uses it (e.g. now the code uses a name that doesn't exist anymore).

FreeBSD 12...11,

I was actually referring to changes like this, with happened when updating libc to support FreeBSD 11 instead of FreeBSD 10 (EOL): #1222

Note how the type of struct stack_t::ss_sp changed from *mut c_char to *mut c_void. This did not broke anything because nobody was using it, but if the standard library were passing that struct field to a function taking a *mut c_char, then this change would have broken it. I don't understand how replacing libc with a different one that has the same backwards incompatible change could avoid breakage in similar cases to this one, were a type changes.

[myfreeweb]

please don't rewrite every OS"s C library in Rust. That's basically what Go tried, and it hasn't gone smoothly. It turns out there's some hard stuff in the C libraries.

Also the libc is the stable and public API on most systems. (Systems that don't have a penguin logo, that is.)

And doing what Go does is a massive pain for less popular CPU architecture + OS combos.

forked libc

hm — rustc already comes with a libc, but it's private and everyone is forced to redownload it with cargo. Switching everything to the libc distributed with rustc, such that it's just like libcore/libstd, will automatically achieve "forked libc".

[raphaelcohn]

PRs welcome I guess
Life is too short! It's either a complete re-working of the entire structure, with a proper focus on complete implementation of everything in libc for every major variant, or nothing...

every time I've used it, it brings me to the relevant source code
Which is often difficult to truly get an understanding of with a lot of cross platform [#cfg]. Using the file system to visually see high-level difference is often superior.

@asomers : It's not really that hard to write most of a libc - just an awful lot of work. If one is looking to only support pure-Rust, or nearly pure Rust implementations, or modern 4.0+ (say) Linux, then a stripped back libc-alike is quite do-able for Linux. That said, this is definitely a 'mad, mad' option. However, I've always been very doubtful that the wins are worthwhile versus the effort. There might be some small inlining gains, but, for the sort of network server software I tend to write, the only real gain is strip away the (very thin) layers before making a syscall - and I try to avoid syscalls.

@myfreeweb : agreed - it's the public, stable API on old-fashioned OSes (;-). And it's a pain in the neck, especially for cross-compilation.

[gnzlbg]

If one is looking to only support pure-Rust, or nearly pure Rust implementations, or modern 4.0+ (say) Linux, then a stripped back libc-alike is quite do-able for Linux.

This issue is about finding a solution to breaking changes in platform APIs, yet Linux is one of the few OSes that never break their API.

I don't know what problem re-implementing libc in Rust for Linux would solve, but I don't see how it could solve the one being discussed here, since Linux does not have it AFAICT.

[raphaelcohn]

@gnzblg: Agreed; just re-implementing libc in Rust for Linux is not going to solve the problem being discussed here. I'm happy to stop discussing it - it's a distraction.