[tracking] solo5 (virtio) on FreeBSD bhyve #61

Closed
hannesm opened this Issue Jul 6, 2016 · 15 comments

Comments

Projects
None yet
3 participants
@hannesm
Contributor

hannesm commented Jul 6, 2016

status is: it works, but still needs some manual patches. this uses clang instead of gcc. the instructions are documented here. I haven't tested more than console yet.

Updates:

--- old stuff below ---

This issue is to track progress of upstreaming this into the solo5 mainline. The biggest issue are include files.

  • libgcc.a is not needed to get linked in (it seems to be required on arm only, see #50 and mirage/mirage#544), PR #63
  • solo5 uses nostdinc, but then includes ${CC} -print-file-name=include as system includes. It then includes stddef.h from there, which drags in sys/_types.h on FreeBSD. There are other headers needed, such as osreldate.h (by openlibm). In order to build mirage-entropy-solo5 I needed to get rid of the gcc includes to get the intrinsics shipped with clang. I've no proper idea how to deal with this best: provide our own stddef.h? remove nostdinc? Input is welcome how to solve this (note: the vanilla minios-xen uses nostdinc and does not use any system include (thus builds fine with clang); mirage modified minios drags in stdint.h and I wasn't able to get it working on FreeBSD). e.g. grub completely ships their own header files.
  • ocaml-freestanding: POSIX cp does not have a -d flag commit (which is there to copy sys/time.h and its symlink time.h) -- see posix time.h vs posix sys/time.h
  • ocaml-freestanding: openlibm does use gcc as CC on non-Darwin systems, and needs to be patched (should be upstreamed to contain CC?=cc behaviour) (fixed upstream in openlibm, see mirage/ocaml-freestanding#4)
  • ocaml-freestanding: other Makefile.x86_64 for FreeBSD commit (includes the openlibm fix)
  • ocaml-freestanding: nolibc uses -Wall -Werror, but excludes -Wno-maybe-uninitialized which is gcc only (clang fails on that unknown option) -- a compiler-specific Makefile rule may solve that (can we fix the code instead?)
  • mirage: need to use ld from ports (thus /usr/local/bin/ld), the system provided one is too old to cope with -z max-page-size=0x1000 commit

comments, especially on how to sanely deal with stddef.h and friends (machine/_types.h machine/endian.h sys/_types.h sys/cdefs.h x86/_types.h x86/endian.h osreldate.h) would be highly appreciated!

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Jul 7, 2016

Member

Regarding include files, we can't remove -nostdinc as the results of including anything from /usr/include are unpredictable. I think the best option is to lift musl stddef.h, stdint.h, stdarg.h and the infrastructure needed to build their dependencies (arch/x86_64/bits/alltypes.h.in, tools/mkalltypes.sed, ...) and ship them with Solo5 directly. Musl supports clang so these should work fine (http://wiki.musl-libc.org/wiki/Supported_Platforms). Any objections to this approach? (ping @djwillia)

Regarding the ocaml-freestanding changes, there is a configure.sh which generates Makeconf at build time, so you could use this to detect if building on FreeBSD and/or clang and set the appropriate flags and asmrun Makefile there.

As for the dependency on ld from ports, not much we can do about that in the short term, other than:

  1. Add Makeconf / configure.sh-style infrastructure to Solo5 and try and look for a good enough ld in "likely" places, failing if none was found.
  2. Add pkg-config infrastructure to pass the correct ld from solo5-kernel-virtio through to mirage.
Member

mato commented Jul 7, 2016

Regarding include files, we can't remove -nostdinc as the results of including anything from /usr/include are unpredictable. I think the best option is to lift musl stddef.h, stdint.h, stdarg.h and the infrastructure needed to build their dependencies (arch/x86_64/bits/alltypes.h.in, tools/mkalltypes.sed, ...) and ship them with Solo5 directly. Musl supports clang so these should work fine (http://wiki.musl-libc.org/wiki/Supported_Platforms). Any objections to this approach? (ping @djwillia)

Regarding the ocaml-freestanding changes, there is a configure.sh which generates Makeconf at build time, so you could use this to detect if building on FreeBSD and/or clang and set the appropriate flags and asmrun Makefile there.

As for the dependency on ld from ports, not much we can do about that in the short term, other than:

  1. Add Makeconf / configure.sh-style infrastructure to Solo5 and try and look for a good enough ld in "likely" places, failing if none was found.
  2. Add pkg-config infrastructure to pass the correct ld from solo5-kernel-virtio through to mirage.
@djwillia

This comment has been minimized.

Show comment
Hide comment
@djwillia

djwillia Jul 7, 2016

Member

@mato @hannesm I agree; it seems the best option is to put those headers into the tree.

At the risk of sounding ignorant, if we are properly providing a build container (e.g., like mato's one from unikernel-runner), do we really need to solve compilation-on-multiple-platform issues? I see the importance of having the resultant binaries and images run on other systems, but not the build? The only possible upside I see is that building on multiple platforms may expose more bugs or weirdness that should be fixed regardless. Am I missing something? Has Docker not yet been made to work in a stable way with FreeBSD?

Member

djwillia commented Jul 7, 2016

@mato @hannesm I agree; it seems the best option is to put those headers into the tree.

At the risk of sounding ignorant, if we are properly providing a build container (e.g., like mato's one from unikernel-runner), do we really need to solve compilation-on-multiple-platform issues? I see the importance of having the resultant binaries and images run on other systems, but not the build? The only possible upside I see is that building on multiple platforms may expose more bugs or weirdness that should be fixed regardless. Am I missing something? Has Docker not yet been made to work in a stable way with FreeBSD?

@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Jul 7, 2016

Contributor

@djwillia while djwillia@5084a97 is mainly cosmetic, mato/ocaml-freestanding@b0c34ec is a nice example why supporting multiple build platforms is a good idea. Detecting bugs, be compatible to some POSIX standard...

clearly you can depend on having some docker installation to build solo5. I appreciate more minimalistic approaches (which do not include dependency on yet another PL runtime (Go) etc.) for building a piece of software, esp since getting solo5 to build on FreeBSD (or other POSIX systems) is not a big pain. I'd not want to add a dependency onto autoconf and friends, but some conditionals based on uname -s during build is IMHO fine. while I have not tried docker (did not have the demand for it), there is a port to FreeBSD AFAIK.

Contributor

hannesm commented Jul 7, 2016

@djwillia while djwillia@5084a97 is mainly cosmetic, mato/ocaml-freestanding@b0c34ec is a nice example why supporting multiple build platforms is a good idea. Detecting bugs, be compatible to some POSIX standard...

clearly you can depend on having some docker installation to build solo5. I appreciate more minimalistic approaches (which do not include dependency on yet another PL runtime (Go) etc.) for building a piece of software, esp since getting solo5 to build on FreeBSD (or other POSIX systems) is not a big pain. I'd not want to add a dependency onto autoconf and friends, but some conditionals based on uname -s during build is IMHO fine. while I have not tried docker (did not have the demand for it), there is a port to FreeBSD AFAIK.

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Jul 7, 2016

Member

@djwillia The Docker on FreeBSD port seems dormant (https://github.com/kvasdopil/docker/tree/freebsd-compat). In any case, while providing a dockerized build is great I don't think we should make Docker a hard dependency. And, as both you and @hannesm agree, building on multiple platforms tends to expose bugs so is worth the trouble :-)

Member

mato commented Jul 7, 2016

@djwillia The Docker on FreeBSD port seems dormant (https://github.com/kvasdopil/docker/tree/freebsd-compat). In any case, while providing a dockerized build is great I don't think we should make Docker a hard dependency. And, as both you and @hannesm agree, building on multiple platforms tends to expose bugs so is worth the trouble :-)

@djwillia

This comment has been minimized.

Show comment
Hide comment
@djwillia

djwillia Jul 7, 2016

Member

@hannesm @mato Thanks for the explanations; makes sense to me!

Member

djwillia commented Jul 7, 2016

@hannesm @mato Thanks for the explanations; makes sense to me!

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Jul 19, 2016

Member

@hannesm I've looked at ways of resolving the header file issue and unfortunately it's more complicated than I originally thought.

There are two groups of header files involved:

  1. Standard C definitions: stdint.h, stddef.h, stdarg.h, ... gcc bundles these, clang does not.
  2. CPU intrinsics: cpuid.h, x86intrin.h, ... gcc and clang both bundle these and they are relied on by (at least) the nocrypto code.

We can supply our own headers for "group 1", but not for "group 2" since these are compiler specific. However, I can't find any practical way of making it work:

  1. With gcc, both groups are in the directory we get via gcc -print-file-name=include. So we can't get one without the other, hence shipping our own headers is kind of pointless.

  2. With clang (on my FreeBSD 10.3-RELEASE test VM, clang 3.4.1) the "group 2" files live in /usr/include/clang/3.4.1. According to the man page, clang has a -nostdlibinc option which is documented as:

    Do not search the standard system directories for include files, but do search compiler builtin include directories.

    However, this does not actually work -- I tried it with a test program and the compiler cannot find cpuid.h with -nostdlibinc. I tried various other options to clang but could not find one that would point me to the "builtin" include directory.

Conclusion: In the interests of getting initial versions of the Mirage/Solo5 packages published, I'd like to postpone this work, keep the dependency on the libgcc headers and revisit the clang situation later.

Member

mato commented Jul 19, 2016

@hannesm I've looked at ways of resolving the header file issue and unfortunately it's more complicated than I originally thought.

There are two groups of header files involved:

  1. Standard C definitions: stdint.h, stddef.h, stdarg.h, ... gcc bundles these, clang does not.
  2. CPU intrinsics: cpuid.h, x86intrin.h, ... gcc and clang both bundle these and they are relied on by (at least) the nocrypto code.

We can supply our own headers for "group 1", but not for "group 2" since these are compiler specific. However, I can't find any practical way of making it work:

  1. With gcc, both groups are in the directory we get via gcc -print-file-name=include. So we can't get one without the other, hence shipping our own headers is kind of pointless.

  2. With clang (on my FreeBSD 10.3-RELEASE test VM, clang 3.4.1) the "group 2" files live in /usr/include/clang/3.4.1. According to the man page, clang has a -nostdlibinc option which is documented as:

    Do not search the standard system directories for include files, but do search compiler builtin include directories.

    However, this does not actually work -- I tried it with a test program and the compiler cannot find cpuid.h with -nostdlibinc. I tried various other options to clang but could not find one that would point me to the "builtin" include directory.

Conclusion: In the interests of getting initial versions of the Mirage/Solo5 packages published, I'd like to postpone this work, keep the dependency on the libgcc headers and revisit the clang situation later.

@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Jul 19, 2016

Contributor

thanks for investigating. regarding 2: clang -print-file-name=include returns the directory for group 2 (on my FreeBSD-CURRENT /usr/bin/../lib/clang/3.8.0/include, there is no longer any /usr/include/clang)

Contributor

hannesm commented Jul 19, 2016

thanks for investigating. regarding 2: clang -print-file-name=include returns the directory for group 2 (on my FreeBSD-CURRENT /usr/bin/../lib/clang/3.8.0/include, there is no longer any /usr/include/clang)

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Jul 19, 2016

Member

Ok, so that's changed. On 10.3-RELEASE it returns /usr/lib/include. Can you check if the following compiles with -nostdlibinc on CURRENT?

#include <cpuid.h>

int main(int argc, char *argv[])
{
    int max = __get_cpuid_max (0, &max);
    return max;
}
Member

mato commented Jul 19, 2016

Ok, so that's changed. On 10.3-RELEASE it returns /usr/lib/include. Can you check if the following compiles with -nostdlibinc on CURRENT?

#include <cpuid.h>

int main(int argc, char *argv[])
{
    int max = __get_cpuid_max (0, &max);
    return max;
}
@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Jul 19, 2016

Contributor

that works fine here (clang 3.8.0):

] cc -nostdlibinc -o foo foo.c
foo.c:5:35: warning: passing 'int *' to parameter of type 'unsigned int *' converts between pointers to
      integer types with different sign [-Wpointer-sign]
    int max = __get_cpuid_max (0, &max);
                                  ^~~~
/usr/bin/../lib/clang/3.8.0/include/cpuid.h:181:74: note: passing argument to parameter '__sig' here
static __inline int __get_cpuid_max (unsigned int __level, unsigned int *__sig)
                                                                         ^
1 warning generated.
] ./foo
] echo $?
20
Contributor

hannesm commented Jul 19, 2016

that works fine here (clang 3.8.0):

] cc -nostdlibinc -o foo foo.c
foo.c:5:35: warning: passing 'int *' to parameter of type 'unsigned int *' converts between pointers to
      integer types with different sign [-Wpointer-sign]
    int max = __get_cpuid_max (0, &max);
                                  ^~~~
/usr/bin/../lib/clang/3.8.0/include/cpuid.h:181:74: note: passing argument to parameter '__sig' here
static __inline int __get_cpuid_max (unsigned int __level, unsigned int *__sig)
                                                                         ^
1 warning generated.
] ./foo
] echo $?
20
@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Jul 21, 2016

Contributor

I updated the task list at the top!

Contributor

hannesm commented Jul 21, 2016

I updated the task list at the top!

@hannesm hannesm referenced this issue Sep 1, 2016

Closed

[Tracking] Solo5 tasks for Mirage 3.0 release #82

8 of 8 tasks complete
@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Oct 14, 2016

Member

@hannesm I got tired of having to hand-apply hacks to build and test virtio on FreeBSD, so I've integrated most of your changes:

  • #102 has the Solo5 parts. Uses a variant of the /tmp/myinclude approach to make things work.
  • mirage/ocaml-freestanding#9 has the rest.
    Could you review/test these and let me know what you think? I've tested on 11.0-RELEASE and partially on 10.3-RELEASE and all seems fine.

The following parts are still missing for an "out of the box" experience:

  • conf-pkg-config needs to gain a depext on pkgconf for FreeBSD.
  • Makefile generation in mirage should be modified to set the default LD for Solo5 targets from the output of pkg-config solo5-kernel-virtio --variable=ld.

If you could handle the above I'd appreciate it!

Member

mato commented Oct 14, 2016

@hannesm I got tired of having to hand-apply hacks to build and test virtio on FreeBSD, so I've integrated most of your changes:

  • #102 has the Solo5 parts. Uses a variant of the /tmp/myinclude approach to make things work.
  • mirage/ocaml-freestanding#9 has the rest.
    Could you review/test these and let me know what you think? I've tested on 11.0-RELEASE and partially on 10.3-RELEASE and all seems fine.

The following parts are still missing for an "out of the box" experience:

  • conf-pkg-config needs to gain a depext on pkgconf for FreeBSD.
  • Makefile generation in mirage should be modified to set the default LD for Solo5 targets from the output of pkg-config solo5-kernel-virtio --variable=ld.

If you could handle the above I'd appreciate it!

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Oct 14, 2016

Member

Additionally, the following two commits are required to build gmp-freestanding and zarith-freestanding on FreeBSD:

Member

mato commented Oct 14, 2016

Additionally, the following two commits are required to build gmp-freestanding and zarith-freestanding on FreeBSD:

@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Oct 14, 2016

Contributor

thx @mato -- the zarith-freestanding and gmp-freestanding have just been merged to opam-repository! :) (see ocaml/opam-repository#7614)

Contributor

hannesm commented Oct 14, 2016

thx @mato -- the zarith-freestanding and gmp-freestanding have just been merged to opam-repository! :) (see ocaml/opam-repository#7614)

@mato

This comment has been minimized.

Show comment
Hide comment
@mato

mato Oct 17, 2016

Member

I've pushed some more fixes to both PRs. With these (and ocaml/opam-repository #7627) I can now build the whole stack (well, static_website_tls) and the built binary runs fine on 11.0-RELEASE.

I've also been testing on 10.3-RELEASE, things work 80% of the time there, unfortunately mirage-entropy-solo5 will not build due to clang -nostdlibinc missing headers for the CPU builtins. I'm not going to try and fix this, suggest to ask people to just use 11.0-RELEASE or newer.

@hannesm So, I think it's OK to merge #102 and mirage/ocaml-freestanding#9 -- if there are issues with -CURRENT (I hope not!) we can fix those separately.

Member

mato commented Oct 17, 2016

I've pushed some more fixes to both PRs. With these (and ocaml/opam-repository #7627) I can now build the whole stack (well, static_website_tls) and the built binary runs fine on 11.0-RELEASE.

I've also been testing on 10.3-RELEASE, things work 80% of the time there, unfortunately mirage-entropy-solo5 will not build due to clang -nostdlibinc missing headers for the CPU builtins. I'm not going to try and fix this, suggest to ask people to just use 11.0-RELEASE or newer.

@hannesm So, I think it's OK to merge #102 and mirage/ocaml-freestanding#9 -- if there are issues with -CURRENT (I hope not!) we can fix those separately.

@hannesm

This comment has been minimized.

Show comment
Hide comment
@hannesm

hannesm Oct 18, 2016

Contributor

thanks (especially to @mato) we're finally in a state where this all works (more or less) out of the box, the only missing bit is the LD define, which is PR'ed in mirage/mirage#623 and was just merged :)

Contributor

hannesm commented Oct 18, 2016

thanks (especially to @mato) we're finally in a state where this all works (more or less) out of the box, the only missing bit is the LD define, which is PR'ed in mirage/mirage#623 and was just merged :)

@hannesm hannesm closed this Oct 18, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment