Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement libc in zig #514

Open
andrewrk opened this Issue Oct 1, 2017 · 14 comments

Comments

Projects
None yet
4 participants
@andrewrk
Copy link
Member

andrewrk commented Oct 1, 2017

If Zig ships with a libc implementation, it seals the deal for streamlined cross platform compilation. It improves the situation for native compilation as well. When a zig project depends on a C library such as GLFW, one could package it up with zig build system to build with zig (See #490). This accomplishes several things:

  • A single zig build from the root project on any platform will fetch the project's dependencies and perform a deterministic build using only zig and no system dependencies.
  • Zig can be intelligent about how it links dependencies together. It can link a bunch of .o files together instead of doing dynamic linking, where only 1 of the .o files has the libc implementation in it. So any code shared by zig std library and the libc implementation is actually shared in the binary, including stdio buffers and such. The default allocator can be shared between zig std lib and malloc/free.
  • This build would have no native dependencies, so cross compiling works perfectly out-of-the-box, even though you have C library dependencies that depend on libc.

Our goal is for developers to reach for zig instead of gcc, clang, or msvc, even if they intend to use it only to compile C code.

For linux, we can port musl. A lot of our stdlib is ported from there anyway - a lot of this code would be shared between zig standard library and the libc implementation. We should also look at glibc, for example for the getpwnam function, which in musl reads /etc/passwd but in glibc has an extension for a more advanced user account system.

For windows it might be worth looking into newlib-cygwin, reactos libc, and wine libc.

For macos, we always link dynamically against libSystem which has libc in it because this is the stable syscall interface. When cross compiling we allow functions to be unresolved that come from libc. We still need to ship .h files with appropriate constant values, however. I think the .h files from apple might be BSD licensed and thus we could redistribute those.

@pluto439

This comment has been minimized.

Copy link

pluto439 commented Nov 14, 2017

On windows, from what part of libc should people start?

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Nov 14, 2017

Can you elaborate on the question?

@pluto439

This comment has been minimized.

Copy link

pluto439 commented Nov 14, 2017

Do you need all libc, or only some parts of it? Which parts are more important right now?

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Nov 14, 2017

The goal is to provide API compatibility for existing C projects. We can let that drive the use case. So to figure out what is more important, we would try to, for example, build ffmpeg against a zig-libc.

@pluto439

This comment has been minimized.

Copy link

pluto439 commented Nov 14, 2017

I'd start with libjpeg, but I need exceptions for that. Or at least setjmp/longjmp.

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Nov 14, 2017

You can implement setjmp/longjmp with inline assembly.

@tiehuis

This comment has been minimized.

Copy link
Member

tiehuis commented Mar 22, 2018

For anyone interested in this, I've started a simple repository which should give you some ideas here. Anyone reading, feel free to use this as a base or as inspiration.
https://github.com/tiehuis/zligc

I won't be doing much implementation here (too many things I already have to work on) but I may use this to figure out some zig -> c questions and some missing build system details needed in zig to accomodate the required goals.

This was referenced Mar 27, 2018

andrewrk added a commit that referenced this issue Apr 11, 2018

@suirad

This comment has been minimized.

Copy link
Contributor

suirad commented Oct 6, 2018

@andrewrk would you mind making a zlibc repo under the ziglang account that people could coordinate on to accomplish this? You could probably start it with a fork of @tiehuis' repo if you want.
I know you probably already have plenty to manage with zig itself, so you could probably recruit a mod or two to manage the repo.

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Feb 25, 2019

@suirad sorry, I missed your comment above. The libc code is going to live in this repository itself; contributions can be done via pull requests. I'll get the effort started so that people can see the pattern.

Now that we have #490 done, this issue of having libc is even more valuable, and easier to implement. How it's going to work is that Zig will have a set of supported libcs corresponding to a target.

  • --target-os linux --target-environ gnu (or other gnu environs) will be glibc, and it will be a dynamic dependency, the only kind supported by glibc. Zig will be able to build glibc's startup files from source (crtn.o, crtbegin.o, etc) and provide .h files. Zig does not need implementations of libc functions since it is dynamically loaded.
  • --target-os linux --target-environ musl (or other musl environs) will be musl's .h files. It will be a static dependency. Some of musl's source code can be imported directly such as their startup files. Other implementations of libc functions will be shared with the Zig standard library - this will prevent code bloat. For example the math functions will share implementations.
  • --target-os windows --target-environ msvc I think for this one we will use newlib/wine source code. It will be a static dependency.
  • --target-os windows --target-environ gnu (or other gnu environs) - this is mingw libc. Same idea as musl. Static dependency. Build from source.
  • --target-os freebsd (not sure what the default target environ is) - this is similar to glibc for linux. Zig will be able to build the startup files from source, but the implementations will be resolved at runtime.
  • --target-os macosx -libSystem actually does not even require any startup files. Brilliant.
  • --target-os netbsd same idea as freebsd

Once this is done, it raises the bar for Tier 1. A target can only be Tier 1 if Zig can cross-compile link against its libc.

Zig will no longer look for the native libc and try to build against it. zig libc will be the only way to do that. And unless a --libc foo.txt argument is provided to use a custom libc, Zig will use one from the bulleted list above; never the native one. This makes Zig 100% free of system dependencies, which makes it a true cross compiler.

andrewrk added a commit that referenced this issue Mar 7, 2019

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Mar 8, 2019

I've done a proof of concept of this with glibc and it's merged into master. (The CI build is broken but I have a fix coming shortly)

There's a new tool libc/process_headers.zig and a wiki page documenting how to maintain the various libcs.

Here are some screenshots on Windows, Linux, macOS:

cross-glibc-windows
cross-glibc-macos
cross-glibc-linux

So at this point it's a checklist:

  • musl libc - this one we will do the headers thing, but try to implement most of the functions in Zig. Some of the functions we can build from C source - that's fine. We can always do a piecemeal port.
  • Windows libc (newlib? wine?) - same idea as musl - get the headers from somewhere, but the implementation should be shared with musl implementation, with perhaps some .c code to augment it.
  • mingw libc / cygwin libc - these will probably be their own issue. Probably low priority. Why target these when you can target Windows directly?
  • macosx libc - we don't need any startup files for this one, but we do need headers.
  • FreeBSD libc - follow glibc's example
  • NetBSD libc - follow glibc's example
  • OpenBSD libc - follow glibc's example

@andrewrk andrewrk referenced this issue Mar 8, 2019

Open

Tier 1 FreeBSD Support for x86_64 #1759

5 of 7 tasks complete
@suirad

This comment has been minimized.

Copy link
Contributor

suirad commented Mar 8, 2019

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time. That should be added to the checklist.

The reason why it should be exposed and not assumed based on OS, is because some platforms are able to use multiple types, such as linux with musl & glibc.

@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Mar 8, 2019

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time.

That's @import("builtin").abi, which is specified at compile time with -target <arch><subarch>-<os>-<abi>.

andrewrk added a commit that referenced this issue Mar 12, 2019

ability to build musl from source
bundles musl 1.1.21

See #514
@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Mar 13, 2019

Musl is done
Screenshot_2019-03-12_17-24-17

Repeating this here:
Currently zig builds musl from source with minimal modifications. The next step will be to also include std/special/libc.zig (which I have not started yet) as the root source file when creating libc.a for musl. And then slowly, we can start deleting C source files from the musl that zig bundles, and porting the code to zig instead.

Another nice optimization/research area would be to not have a separate .a file for libc, but actually export some additional functions from the same object file as the root zig source file. In this way the libc and the zig code could share implementations and shave off bloat. This would probably be helped a lot by an incremental compiler.

@andrewrk andrewrk referenced this issue Mar 13, 2019

Open

test coverage for all the libc targets #2058

0 of 43 tasks complete
@andrewrk

This comment has been minimized.

Copy link
Member Author

andrewrk commented Mar 13, 2019

For Windows, someone pointed me to MidiPix which could be another source of inspiration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.