New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Porting ukvm to OpenBSD vmm APIs #206

Closed
adamsteen opened this Issue Jul 15, 2017 · 64 comments

Comments

Projects
None yet
6 participants
@adamsteen
Copy link
Contributor

adamsteen commented Jul 15, 2017

Hi

I am raising this issue after a post on Unikernel Devel, @hannesm said it would be preferable to discuss the situation here.

I am in the process of porting Solo5 to OpenBSD. I have successfully built the virtio kernel and tests using clang.

But when attempting to boot an image (created via mkimage on an alpine linux install) it fails with.

Booting from Hard Disk...

Booting from 0000:7c00

SYSLINUX 6.03 EDD 2014-10-06 Copyright (C) 1994-2014 H. Peter Anvin et al
enter handle_12:
   a=00000000  b=00000000  c=00000000  d=00000000 ds=0000 es=0000 ss=0000
  si=00000000 di=00000000 bp=00000000 sp=00007b2e cs=0000 ip=8c2d  f=0000
enter handle_12:
   a=00000200  b=00000000  c=00000000  d=00000000 ds=0000 es=0000 ss=0000
  si=00000000 di=00000000 bp=00000000 sp=00007b2e cs=0000 ip=8c2d  f=0000

SYSLINUX 6.03 2014-10-06 Copyright (C) 1994-2014 H. Peter Anvin et al
Loading unikernel.bin... ok

enter handle_12:
   a=00000000  b=00000000  c=00000000  d=00000000 ds=0000 es=0000 ss=0000
  si=00000000 di=00000000 bp=00000000 sp=00007b0e cs=0000 ip=8c2d  f=0000
Invalid Multiboot image: neither ELF header nor a.out kludge found

boot:

I expect this is due to an old binutils.
The output of "ld --version"

GNU ld version 2.17
Copyright 2005 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License.  This program has absolutely no warranty.

The output of readelf -S for both Linux and OpenBSD (these are different) is as follows
(built and readelf on the specified platform)

readelf -S test_hello.virtio

OpenBSD

There are 19 section headers, starting at offset 0x12f1d0:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000000100000  00100000
       0000000000009169  0000000000000000  AX       0     0     16
  [ 2] .rodata           PROGBITS         000000000010a000  0010a000
       0000000000000cdd  0000000000000000   A       0     0     16
  [ 3] .eh_frame         X86_64_UNWIND    000000000010ace0  0010ace0
       0000000000000e38  0000000000000000   A       0     0     8
  [ 4] .data             PROGBITS         000000000010c000  0010c000
       0000000000004008  0000000000000000  WA       0     0     4096
  [ 5] .bss              NOBITS           0000000000111000  00110008
       0000000000006960  0000000000000000  WA       0     0     16
  [ 6] .debug_aranges    PROGBITS         0000000000000000  00110008
       0000000000000060  0000000000000000           0     0     1
  [ 7] .debug_pubnames   PROGBITS         0000000000000000  00110068
       0000000000000f56  0000000000000000           0     0     1
  [ 8] .debug_info       PROGBITS         0000000000000000  00110fbe
       00000000000096ae  0000000000000000           0     0     1
  [ 9] .debug_abbrev     PROGBITS         0000000000000000  0011a66c
       0000000000001abc  0000000000000000           0     0     1
  [10] .debug_line       PROGBITS         0000000000000000  0011c128
       00000000000052ca  0000000000000000           0     0     1
  [11] .debug_str        PROGBITS         0000000000000000  001213f2
       0000000000001891  0000000000000001  MS       0     0     1
  [12] .debug_loc        PROGBITS         0000000000000000  00122c83
       0000000000009675  0000000000000000           0     0     1
  [13] .debug_macinfo    PROGBITS         0000000000000000  0012c2f8
       0000000000000017  0000000000000000           0     0     1
  [14] .debug_ranges     PROGBITS         0000000000000000  0012c30f
       0000000000001c90  0000000000000000           0     0     1
  [15] .debug_pubtypes   PROGBITS         0000000000000000  0012df9f
       0000000000001166  0000000000000000           0     0     1
  [16] .shstrtab         STRTAB           0000000000000000  0012f105
       00000000000000c6  0000000000000000           0     0     1
  [17] .symtab           SYMTAB           0000000000000000  0012f690
       0000000000001770  0000000000000018          18   214     8
  [18] .strtab           STRTAB           0000000000000000  00130e00
       0000000000000b4a  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

Linux (Alpine) (ld version: 2.28)

There are 18 section headers, starting at offset 0x43cb8:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000000100000  00001000
       000000000000856f  0000000000000000  AX       0     0     16
  [ 2] .rodata           PROGBITS         0000000000109000  0000a000
       0000000000000bd8  0000000000000000   A       0     0     8
  [ 3] .eh_frame         PROGBITS         0000000000109bd8  0000abd8
       0000000000000fd8  0000000000000000   A       0     0     8
  [ 4] .got              PROGBITS         000000000010b000  0000c000
       0000000000000018  0000000000000008  WA       0     0     8
  [ 5] .data             PROGBITS         000000000010c000  0000d000
       0000000000004120  0000000000000000  WA       0     0     4096
  [ 6] .bss              NOBITS           0000000000111000  00011120
       0000000000006a30  0000000000000000  WA       0     0     32
  [ 7] .comment          PROGBITS         0000000000000000  00011120
       000000000000001a  0000000000000001  MS       0     0     1
  [ 8] .debug_aranges    PROGBITS         0000000000000000  00011140
       00000000000004b0  0000000000000000           0     0     16
  [ 9] .debug_info       PROGBITS         0000000000000000  000115f0
       000000000000cf22  0000000000000000           0     0     1
  [10] .debug_abbrev     PROGBITS         0000000000000000  0001e512
       0000000000002ab0  0000000000000000           0     0     1
  [11] .debug_line       PROGBITS         0000000000000000  00020fc2
       0000000000002889  0000000000000000           0     0     1
  [12] .debug_str        PROGBITS         0000000000000000  0002384b
       0000000000001a17  0000000000000001  MS       0     0     1
  [13] .debug_loc        PROGBITS         0000000000000000  00025262
       0000000000018b1a  0000000000000000           0     0     1
  [14] .debug_ranges     PROGBITS         0000000000000000  0003dd7c
       0000000000003c90  0000000000000000           0     0     1
  [15] .symtab           SYMTAB           0000000000000000  00041a10
       0000000000001728  0000000000000018          16   211     8
  [16] .strtab           STRTAB           0000000000000000  00043138
       0000000000000ad4  0000000000000000           0     0     1
  [17] .shstrtab         STRTAB           0000000000000000  00043c0c
       00000000000000a5  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

There is no newer port of Binutils on OpenBSD so switch to a newer version is not an easy option. In the mean time i have played with a x86_64-elf cross compiler (gcc 4.9 and bintutils 2.28) and have successfully build and booted virtio kernels.

Booting Images build on OpenBSD requires converting them using mkimage on a linux vm.

VMM boot's other images (Linux for example) using a SeaBIOS image, i expect i would need to find something like "grub2-bhyve", but i would rather expect my energy and time working on a ukvm port.

Looking at the current situation on OpenBSD i expect for now the best course of action would be to build kernels with a x86_64-elf cross compiler and work towards a native port of ukvm to OpenBSD, but would prefer to build the kernels with a OpenBSD tool chain, if possible.

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 17, 2017

Hi @adamsteen,

I expect this is due to an old binutils.
The output of "ld --version"

That's probably the case. What happens if you use the system ld as opposed to one from a binutils port?

readelf -S test_hello.virtio

Can you also post the output of readelf -l?

Looking at the current situation on OpenBSD i expect for now the best course of action would be to build kernels with a x86_64-elf cross compiler and work towards a native port of ukvm to OpenBSD, but would prefer to build the kernels with a OpenBSD tool chain, if possible.

Building using a cross compiler is certainly the most straightforward way to get something working. Having said that, OpenBSD must use its own native toolchain to build its kernel, so it should be possible to build unikernels with it.

Toolchain issues aside, it should be fairly straightforward to port ukvm to run on the OpenBSD vmm APIs. (/cc @MagnusS, you were interested in doing a port)

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jul 17, 2017

Hi @mato

What happens if you use the system ld as opposed to one from a binutils port?

that is the system ld.

readelf -l test_hello.virtio

OpenBSD

Elf file type is EXEC (Executable file)
Entry point 0x1000e0
There are 2 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000100000 0x0000000000100000 0x0000000000100000
                 0x0000000000010008 0x0000000000017960  RWE    100000
  NULL           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000         0

 Section to Segment mapping:
  Segment Sections...
   00     .text .rodata .eh_frame .data .bss 
   01     

Linux

Elf file type is EXEC (Executable file)
Entry point 0x100180
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000001000 0x0000000000100000 0x0000000000100000
                 0x000000000000abb0 0x000000000000abb0  R E    0x1000
  LOAD           0x000000000000c000 0x000000000010b000 0x000000000010b000
                 0x0000000000005120 0x000000000000ca30  RW     0x1000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RWE    0x10

 Section to Segment mapping:
  Segment Sections...
   00     .text .rodata .eh_frame 
   01     .got .data .bss 
   02     

OpenBSD must use its own native toolchain to build its kernel, so it should be possible to build unikernels with it.
Will have to investigate this further.

I hope to have a PR soon, but that depends on life.

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 17, 2017

that is the system ld.

Oh my. Yet another case of "let's fork GNU ld at some arbitrary pre-GPLv3 point and maintain it". But, I digress. On FreeBSD we had the same problem and as a workaround for a while we depended on having the binutils port installed and used the ld from that. That approach might also work for you.

 Section to Segment mapping:
  Segment Sections...
   00     .text .rodata .eh_frame .data .bss 
   01     

This is definitely not correct (and is probably caused by the old ld deciding we are building something like an "omagic" binary). However, I don't think it's enough by itself to cause the kernels to fail to boot since the actual PHDR is marked RWE.

Coming back to your original error with SYSLINUX complaining about the multiboot header, have you verified that the multiboot header is correct in the unikernel? e.g. Via a hexdump, or something like the following in GDB:

$ gdb -q tests/test_hello/test_hello.virtio
Reading symbols from tests/test_hello/test_hello.virtio...done.
(gdb) x /8x _multiboot_header 
0x100000 <_stext>:	0x1badb002	0x00010003	0xe4514ffb	0x00100000
0x100010 <_stext+16>:	0x00100000	0x00111000	0x00118000	0x00100020
@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jul 19, 2017

On FreeBSD we had the same problem and as a workaround for a while we depended on having the binutils port installed and used the ld from that. That approach might also work for you.

I am afraid there is no port. i will just have to keep investigating.

have you verified that the multiboot header is correct in the unikernel?

gdb -q tests/test_hello/test_hello.virtio
(gdb) x /8x _multiboot_header
0x100000 <_multiboot_header>:     0x1badb002  0x00010003  0xe4514ffb  0x00100000
0x100010 <_multiboot_header+16>:  0x00100000  0x00111000  0x00118000  0x00100020

which looks correct? except for '_stext' vs '_multiboot_header'

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 19, 2017

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jul 20, 2017

Yes those symbols are in it!

nm -na test_hello.virtio | egrep '_stext|_etext|_erodata|_end'
00100000 T _stext
00109000 R _etext
0010b000 D _erodata
0010b030 d gdt64_end
00117000 B _end

I am continuing to port ukvm as time permits, and will investigate the linking issues as I can. Is there a better way to view this above information?

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 20, 2017

Is there a better way to view this above information?

nm, objdump and readelf all give you the same information in various formats. I tend to use nm as it's the most straightforward. You can also use gdb if you want to do more involved checks.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jul 28, 2017

Just to provide an update on where things sit for OpenBSD.

I haven't looked at the linking issue yet, was hoping to get ukvm working first.

I have completed what i thought would be the required coding work for porting ukvm to OpenBSD at adamsteen/solo5.

But am now trying to figure out setting up the registers to be compatible with OpenBSD vmm and ukvm, looking at the vmm kernel code it appears OpenBSD is messing with the application supplied register request, still looking into this.

When executing
doas ukvm/ukvm-bin tests/test_hello/test_hello.ukvm
ukvm fails with
ukvm-bin: ukvm_hv_vcpu_loop: vm / vcpu run ioctl failed: Invalid argument
and from the kernel
vmx_handle_exit: unhandled exit 0x80000021 (unknown)
from googling 0x80000021 appears to be a hardware error, but can't find any concrete documentation.

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 28, 2017

But am now trying to figure out setting up the registers to be compatible with OpenBSD vmm and ukvm, looking at the vmm kernel code it appears OpenBSD is messing with the application supplied register request, still looking into this.

I'd suggest asking on the relevant OpenBSD mailing lists about their vmm behaviour, and/or reading the vmm code itself to see what it's doing/expecting.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jul 28, 2017

I have been reading the kernel vmm source and adding debug output to better understand its behavior. I hope to have a chunk of time this weekend to really get into it, but expect I will need to email tech (tech@openbsd.org) next week, i am unsure of the reception i will get.

Thanks for taking the time to respond.

As an aside note do you know if KVM changes the registers ukvm supplies or just passes them straight through?

@mato

This comment has been minimized.

Copy link
Member

mato commented Jul 28, 2017

I have been reading the kernel vmm source and adding debug output to better understand its behavior. I hope to have a chunk of time this weekend to really get into it, but expect I will need to email tech (tech@openbsd.org) next week, i am unsure of the reception i will get.

If nothing else you will be excersizing their vmm APIs in novel and interesting ways which is always a good thing in my book :-)

As an aside note do you know if KVM changes the registers ukvm supplies or just passes them straight through?

It does some massaging of the shadow registers (struct x86_sreg in ukvm terminology) and CPU control registers (CRx / EFER / TR / LDT mainly) to fit the constraints imposed by the VT hardware, but not much else. IIRC FreeBSD's vmm operates similarly but does less, so I had to tweak some of the common register setup values to work correctly with both. There might be some more information in my commit messages in the changes adding the FreeBSD vmm code.

@djwillia

This comment has been minimized.

Copy link
Member

djwillia commented Jul 28, 2017

The 0x80000021 means an invalid VMCS. It's not a very helpful error that the HW gives you, but I've mostly encountered that with incorrect values in the control registers CR or GDT, so maybe checking those might be a first step?

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Aug 2, 2017

Hi Guys

just wanted to give another update, after discussion with the Mike Larkin, the OpenBSD vmm guru. I now have the hello world test running.

doas ukvm/ukvm-bin tests/test_hello/test_hello.ukvm
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Memory map: 512 MB addressable:
Solo5:     unused @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x106fff)
Solo5:     rodata @ (0x107000 - 0x108fff)
Solo5:       data @ (0x109000 - 0x10dfff)
Solo5:       heap >= 0x10e000 < stack < 0x20000000

**** Solo5 standalone test_hello ****

Hello, World
Command line is: ''
ukvm-bin: ukvm_hv_vcpu_loop: vm / vcpu run ioctl failed: Input/output error

there a still a fair bit togo, a few patches i need to write for OpenBSD and probably more discussion, but its feel like i am over the scariest hump.

cheers for you help

I am off to bed now, but will push my changes to my repo in the morning, but you still need a custom OpenBSD to kernel for things to run.

@mato mato changed the title test_hello.virtio fails to boot when built on OpenBSD (Incorrect linking with ld version 2.17) Porting ukvm to OpenBSD vmm APIs Aug 2, 2017

@mato

This comment has been minimized.

Copy link
Member

mato commented Aug 2, 2017

@adamsteen That's awesome! Could you point to the discussions you have, if they're public? I'd be interested.

On a side note, I've retitled this issue to something more appropriate.

@avsm

This comment has been minimized.

Copy link
Contributor

avsm commented Aug 2, 2017

This is awesome progress! I have several openbsd-current boxes, so happy to try the patches out when I/O works as your next user...

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Aug 3, 2017

@mato
Sorry, it was a private email chain. but i can provide a quick summary.

Before i made any changed to OpenBSD vmm, the following was happening

cr0=0x0000000000000020 (pg cd nw am wp NE et ts em mp pe)
real mode was being requested (CR0.PG and CR0.PE are 0)

cr3=0x0000000000000000 (pwt pcd)
we had no page table (which is correct if you are requesting real mode)

MSR 0 @ 0xffffff039b869000 : 0xc0000080 (EFER),
value=0x0000000000000500 (sce LME LMA nxe)
but long mode is also requested (EFER.LME and EFER.LMA == 1)

resulting in "vmx_handle_exit: unhandled exit 2147483681 (unknown)"

ie unrestricted guest was forcing real mode, but i was still requesting long mode.

I had to comment out where vmm was changing cr0, cr3 and cr4 and have the init values set up as

cr0 = X86_CR0_PE | X86_CR0_PG | X86_CR0_NE
cr3 = X86_PML4_BASE
cr4 = X86_CR4_PAE | X86_CR4_OSFXSR | X86_CR4_OSXMMEXCPT | X86_CR4_VMXE
vrs_msrs[VCPU_REGS_EFER] = EFER_LME | EFER_LMA

ie X86_CR4_VMXE being added to X86_CR4_INIT.

just rereading the X86_CR4_INIT comment, this is going to be interesting, KVM doesn't like this bit being set, FreeBSD vmm doesn't mind and OpenBSD requires it.

@hb9cwp

This comment has been minimized.

Copy link
Contributor

hb9cwp commented Aug 26, 2017

Awesome, I am trying to reproduce what you have achieved so far. Did you have a chance to work on it lately? Thanks.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Aug 26, 2017

Hi @hb9cwp

I am sorry you will not be able reproduce my work so far without a custom OpenBSD kernel.

My fork adamsteen/solo5 passes all the tests and hope to have it merged when i get all my work merged into the OpenBSD mainline.

I have a patch waiting for the OpenBSD 6.3 unlock to Improve the accuracy of the TSC frequency calibration. But there is some changes in sys/arch/amd64/amd64/vmm.c, that will not fly, i am waiting to hear back from Mike Larkin, he had some ideas. My changes break normally functionality of vmm, so there is still a little work todo in the kernel.

Cheers
Adam

@hb9cwp

This comment has been minimized.

Copy link
Contributor

hb9cwp commented Aug 26, 2017

Hi Adam,
Thanks, looking forward to try your diffs and hope you can get them merged with both upstreams after unlock. Just let me know if I can assist with testing.
Best regards,
Rolf

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 26, 2017

I have one last patch to get merged into OpenBSD current (its been a long process)

So i started looking at what needs to happen for a Hello Mirage World.

I am not sure if i should be asking this here or in ocaml-freestanding/mirage.

i figured out i would need to also change ocaml-freestanding and mirage and pin each as i went to use my local versions. (I have no ocaml/opam experience)

I was successfully able to complete
opam pin add solo5-kernel-ukvm git://github.com/adamsteen/solo5
but
opam pin add ocaml-freestanding git://github.com/adamsteen/ocaml-freestanding
fails with

### stderr ###
# In file included from src/common.c:3:
# In file included from /home/asteen/.opam/system/build/ocaml-freestanding.0.2.2/build/openlibm/src/math_private.h:23:
# /home/asteen/.opam/system/build/ocaml-freestanding.0.2.2/build/openlibm/src/types-compat.h:4:10: fatal error: 'stdint.h' file not found
# #include <stdint.h>
#          ^~~~~~~~~~
# 1 error generated.
# gmake[1]: *** [Make.inc:146: src/common.c.o] Error 1
# gmake: *** [Makefile:22: build/openlibm/libopenlibm.a] Error 2

which i am a bit skeptical about due to copying files in configure.sh and think i may have missed something.

Could someone point me in the right direction to let me know who/where to ask?

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Oct 26, 2017

@adamsteen which compiler do you use? for gcc (the Linux case in configure.sh), we copy:

        CC_INCDIR=$(${CC} -print-file-name=include)
        cp -R ${CC_INCDIR}/. ${HOST_INCDIR}

on FreeBSD, where clang is used, there is no cc -print-file-name=include -- thus we're manually copying over some includes files from /usr/include. the set of includes is whatever makes the compilation of ocaml-freestanding and gmp-freestanding happy. you may need to adjust (and find where stdint.h is on OpenBSD, as well as finding the transitive dependent headers).

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 26, 2017

@hannesm OpenBSD uses clang (5)

re: Manually copying files: Could you please point me to where this is done for FreeBSD?

I already do this for solo5, i take it, i also need to do this for ocaml-freestanding and gmp-freestanding?

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Oct 26, 2017

@adamsteen sorry for being ambiguous - solo5 is the only library installing header files, which are used by ocaml-freestanding and gmp-freestanding. The copying is done in the configure which you modified for OpenBSD. does /usr/include/stdint.h exist on OpenBSD, and is it copied to ~/.opam/XX/include/solo5-kernel-ukvm/include/host/?

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 26, 2017

@hannesm
does /usr/include/stdint.h exist on OpenBSD: Yes
is it copied to ~/.opam/XX/include/solo5-kernel-ukvm/include/host/? Yes, XX is system for me.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 26, 2017

@hannesm

Do I need to tell ocaml-freestanding to look in ~/.opam/XX/include/solo5-kernel-ukvm/include/host/?

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Oct 26, 2017

@adamsteen ocaml-freestanding gets the include directory via pkg-config, there should be a cflags entry in ~/.opam/system/lib/pkgconfig/solo5-kernel-ukvm.pc with: Cflags: -nostdlibinc -ffreestanding -mno-red-zone -isystem ${includedir}/host -I${includedir} (where includedir=${prefix}/include/solo5-kernel-ukvm/include). you can check what (once you set PKG_CONFIG_PATH appropriately) pkg-config --cflags solo5-kernel-ukvm reports on your OpenBSD.

@mato

This comment has been minimized.

Copy link
Member

mato commented Oct 26, 2017

If you want to isolate at which point the problem starts occuring, try adding #include <stdint.h> to one of the Solo5 test programs; if that doesn't work then there's something wrong with the clang setup. If it does work then the problem is further "down" the stack, presumably in the ocaml-freestanding pkg-config bits.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 27, 2017

@mato I did a quick test with test_hello and that compiled fine

@hannesm
is looks like the output of pkg-config may be the problem. see below

on OpenBSD

$ export PKG_CONFIG_PATH=$(opam config var prefix)/lib/pkgconfig
$ echo $PKG_CONFIG_PATH
/home/asteen/.opam/system/lib/pkgconfig
$ pkg-config --cflags solo5-kernel-ukvm
-fno-pie -fno-stack-protector -nostdlibinc -ffreestanding -mno-red-zone -isystem /../../include/solo5-kernel-ukvm/include/host -I/../../include/solo5-kernel-ukvm/include

vs debian

$ export PKG_CONFIG_PATH=$(opam config var prefix)/lib/pkgconfig
$ echo $PKG_CONFIG_PATH
/home/asteen/.opam/system/lib/pkgconfig
$ pkg-config --cflags solo5-kernel-ukvm
-nostdinc -fno-stack-protector -ffreestanding -mno-red-zone -isystem /home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host -I/home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include

The output of pkg-config --cflags solo5-kernel-ukvm is not including the $PKG_CONFIG_PATH

--EDIT--
actually i didn't look far enough.

my solo5-kernel-ukvm.pc contains prefix=${pcfiledir}/../.. so pcfiledir is empty

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Oct 27, 2017

hmm, I can only report that it work on FreeBSD, where the pc file contains:

prefix=${pcfiledir}/../..
includedir=${prefix}/include/solo5-kernel-ukvm/include
Cflags: -nostdlibinc -ffreestanding -mno-red-zone -isystem ${includedir}/host -I${includedir}

I use pkgconf (http://github.com/pkgconf/pkgconf/) at version 1.3.7

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Oct 27, 2017

@hannesm

$ export PKG_CONFIG_PATH=$(opam config var prefix)/lib/pkgconfig
$ pkg-config --cflags solo5-kernel-ukvm
-fno-pie -fno-stack-protector -nostdlibinc -ffreestanding -mno-red-zone -isystem /../../include/solo5-kernel-ukvm/include/host -I/../../include/solo5-kernel-ukvm/include
$ pkg-conf --cflags solo5-kernel-ukvm
-fno-pie -fno-stack-protector -nostdlibinc -ffreestanding -mno-red-zone -isystem /home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host -I/home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include

That fixed it.

I now have

$ which pkg-config
/home/asteen/.local/bin/pkg-config
$ ls -la /home/asteen/.local/bin/pkg-config
lrwxr-xr-x  1 asteen  asteen  39 Oct 28 05:59 /home/asteen/.local/bin/pkg-config@ -> /home/asteen/.local/pkgconf/bin/pkgconf
$ pkg-config --version
1.3.7

and now opam pin add ocaml-freestanding git://github.com/adamsteen/ocaml-freestanding fails with src/e_acos.c:41:10: fatal error: 'float.h' file not found but i know i HAVE NOT included float.h in solo5 host files.

Thank you sooo much for you help! i screwed up on trying pkg-conf vs pkg-config, nice spot.

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Oct 28, 2017

yeah, this openlibm is pretty big for no good reason... I still would like to get rid of floating points in MirageOS entirely

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Nov 2, 2017

@hannesm I would agree with no floating points in a kernel!

just to keep things updated, I seem to be having problems with the hasgot configure script not picking up the host included, but i haven't had a solid look into it yet, life seems to get in the way.

~/.opam/system/build/ocaml-freestanding.0.2.2/build/ocaml/configure:1143

if sh ./hasgot -i stdint.h; then                                                                                                   │··
   inf "stdint.h found."                                                                                                            │··
   echo "#define HAS_STDINT_H" >> s.h                                                                                               │··
fi  
### stderr ###                                                                                                                          │··
# [...]                                                                                                                                 │··
# In file included from ../byterun/caml/misc.h:24:                                                                                      │··
# ../byterun/caml/config.h:77:26: error: typedef redefinition with different types ('unsigned long' vs '__uint64_t' (aka 'unsigned long │··
long'))                                                                                                                                 │··
# typedef ARCH_UINT64_TYPE uint64_t;                                                                                                    │··
#                          ^                                                                                                            │··
# /home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host/stdint.h:67:21: note: previous definition is here│··
# typedef __uint64_t              uint64_t;                                                                                             │··
#                                 ^                                                                                                     │··
# 9 errors generated.                                                                                                                   │··
# gmake[1]: *** [<builtin>: fail.o] Error 1                                                                                             │··
# gmake: *** [Makefile:40: build/ocaml/asmrun/libasmrun.a] Error 2
@mato

This comment has been minimized.

Copy link
Member

mato commented Nov 2, 2017

@adamsteen Why are you (as far as I can tell from the above) running the OCaml compiler's configure script out of the ocaml-freestanding build process?

To elaborate; the various files in config.in/ are lightly hand-edited copies of those generated for the corresponding OS + architecture by a manual run of configure.sh using the standard host compiler (i.e. without any Solo5 flags). I would start with those used by the FreeBSD port, you shouldn't need to change much in them.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Nov 3, 2017

Morning @mato
I am only running opam pin add ocaml-freestanding git://github.com/adamsteen/ocaml-freestanding -y

I have already changed it but maybe i screwed up there! I hope to have sometime this weekend to look at it.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Nov 25, 2017

to update further, i reset the includes and started again. I think i have found my problem, but just don't know enough to fix it.

i am getting the following error (amount others, but this is the first)

In file included from fail.c:19:
In file included from /home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host/signal.h:38:
In file included from /home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host/sys/signal.h:107:
/home/asteen/.opam/system/lib/pkgconfig/../../include/solo5-kernel-ukvm/include/host/sys/siginfo.h:139:4: error: unknown type name 'pid_t'
                        pid_t   _pid;           /* process ID */
                        ^

but the host includes has the files signal.h, sys/signal.h and sys/types.h
fail.c includes signal.h which intern includes sys/signal.h and sys/types.h, but sys/types.h if protected by a #if

#if __BSD_VISIBLE || __POSIX_VISIBLE || __XPG_VISIBLE
  #include <sys/types.h>
#endif

which is what causes the issue above, unless i am completely miss understanding things.

Does anyone have any ideas on how to proceed?

full error attached.
ocaml-freestanding-47507-c743ac.err

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Dec 1, 2017

Hi All

With the follow two changes ([1],[2]), care of Mike Larkin, my port of
Solo5 [3] successfully passes all the tests on OpenBSD current, you
might have to wait a few days for a snapshot to include the changes.

Should i create a pull request to merge my port or wait until I/we have a
the complete MirageOS working?

I have been working on my port of ocaml-freestanding [4] as the next
step to have MirageOS on OpenBSD, but i am now hitting the limit of my
knowledge, (pure coding i can understand and track down, but build
systems i just don't get), so i have been unable to pin
ocaml-freestanding [4], it fails with "error: unknown type name
'pid_t'", see the error log attached.

pid_t is defined in sys/types.h, which should be in included by
signal.h, but isn't, and thats my problem.

If you would like to test solo5/ukvm on OpenBSD checkout [3] and run
export LD=ld.lld
gmake
tests/setup-tests.sh
tests/run-tests.sh

now to install ocaml-freestanding you will need to do the following

On an up to date OpenBSD current with opam install.

need to install pkgconf 1.3.7 [5]
symbolically link from pkgconf to pkg-config on your path, i install
it to /home/asteen/.local/bin
(ln -s /home/asteen/.local/pkgconf/bin/pkgconf
/home/asteen/.local/bin/pkg-config)

depending on your setup you may need

  • opam init -n
  • eval opam config env
  • export OPAMNOASPCUD=1

export LD=ld.lld
opam install mirage -y ( if not installed )
opam pin add solo5-kernel-ukvm git://github.com/adamsteen/solo5 -y
opam pin add ocaml-freestanding git://github.com/adamsteen/ocaml-freestanding -y

the last command fails with errors as described above., i am not sure
where to proceed from here. see the err attached to the previous comment.

Cheers
Adam

[1] https://marc.info/?l=openbsd-cvs&m=151192357631681&w=2
[2] https://marc.info/?l=openbsd-cvs&m=151192418831778&w=2
[3] https://github.com/adamsteen/solo5
[4] https://github.com/adamsteen/ocaml-freestanding
[5] https://github.com/pkgconf/pkgconf

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Dec 8, 2017

Hi @hannesm @mato
Could you please tell me where ocaml-freestanding is expecting to get siginfo_t, SA_SIGINFO and friends?

I am getting the following errors, and pulling in siginfo.h or signal.h from openbsd just causes more problems.

signals_asm.c:80:1: error: unknown type name 'siginfo_t'
DECLARE_SIGNAL_HANDLER(handle_signal)
^
./signals_osdep.h:129:28: note: expanded from macro 'DECLARE_SIGNAL_HANDLER'
 static void name(int sig, siginfo_t * info, struct sigcontext * context)
                           ^
signals_asm.c:80:1: warning: declaration of 'struct sigcontext' will not be visible outside of this function [-Wvisibility]
./signals_osdep.h:129:53: note: expanded from macro 'DECLARE_SIGNAL_HANDLER'
 static void name(int sig, siginfo_t * info, struct sigcontext * context)
                                                    ^
signals_asm.c:125:5: error: no member named 'sa_sigaction' in 'struct sigaction'
    SET_SIGACT(sigact, handle_signal);
    ^          ~~~~~~
./signals_osdep.h:132:9: note: expanded from macro 'SET_SIGACT'
 sigact.sa_sigaction = (void (*)(int,siginfo_t *,void *)) (name); \
 ~~~~~~ ^
signals_asm.c:125:5: error: unknown type name 'siginfo_t'
./signals_osdep.h:132:38: note: expanded from macro 'SET_SIGACT'
 sigact.sa_sigaction = (void (*)(int,siginfo_t *,void *)) (name); \
                                     ^
signals_asm.c:125:5: error: use of undeclared identifier 'SA_SIGINFO'
./signals_osdep.h:133:20: note: expanded from macro 'SET_SIGACT'
 sigact.sa_flags = SA_SIGINFO
                   ^
1 warning and 4 errors generated.
gmake[1]: *** [<builtin>: signals_asm.o] Error 1
gmake: *** [Makefile:51: build/ocaml/asmrun/libasmrun.a] Error 2

full errors attached
ocaml-freestanding-17226-c743ac.env.txt
ocaml-freestanding-17226-c743ac.err.txt
ocaml-freestanding-17226-c743ac.info.txt
ocaml-freestanding-17226-c743ac.out.txt

@mato

This comment has been minimized.

Copy link
Member

mato commented Dec 12, 2017

I'm afraid I can't quite debug this from the subset of logs here; There's something wrong with the headers in include-host/ provided by solo5-kernel-ukvm and/or the OCaml runtime's usage of them.

The quickest solution would probably be if you could give me access to an OpenBSD box to try and build on; send me an email if you can arrange that.

@avsm Maybe you can help? (With access or debugging)

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Dec 13, 2017

@mato I will see what i can do, all else fails, I will setup you up with a user on my laptop.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Dec 19, 2017

hannesm also commented Why the OpenBSD default ld won't work with solo5. which i also wanted to reference here, to keep all the issues in one place.

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Dec 19, 2017

sorry for being silent here, similar to @mato I atm don't have access to an OpenBSD system (but can ask benno@ or run it somewhere)... certainly real hardware where I can run VMM (and try solo5) would be great, I'll work on this soon. @adamsteen thanks for your work!

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Dec 19, 2017

I have setup mato with an account on my laptop to see if we can get over this last hurdle, i can do the same for you if you would like?

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Dec 19, 2017

@adamsteen thanks, no, that's not needed. If @mato has an account, I'm sure he'll be much faster than me in debugging this (plus I'm travelling today)

@mato

This comment has been minimized.

Copy link
Member

mato commented Dec 19, 2017

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Feb 14, 2018

Hi All

In regards to Why the OpenBSD default ld won't work with solo5. I am not sure we are going to get this working with the default ld GNU ld version 2.17 which is a point before GPLv3. FreeBSD was able to work around this using a port, but none such exists for OpenBSD but ld.lld is availble and seems the best way to go. I am happy to try and work through it, but i think it might be out of my league.

It would be nice to have OpenBSD support in before the next big refactor, I have time this weekend and going to give ocaml-freestanding on OpenBSD another crack.

Looks like i need to do some porting work and bring ocaml upto at least 4.04.2, which version would you recommend?

@hannesm

This comment has been minimized.

Copy link
Contributor

hannesm commented Feb 14, 2018

@adamsteen I recommend OCaml 4.06.0 -- the most recent release which main feature is to use immutable strings by default :)

@mato

This comment has been minimized.

Copy link
Member

mato commented Feb 14, 2018

ld.lld is availble and seems the best way to go. I am happy to try and work through it, but i think it might be out of my league.

Definitely go down this route rather than wasting time trying to fix an ancient broken GNU ld; @hannesm should be able to resurrect the pkg-config stuff for you. In the mean time while developing you can hack around it and force the use of ld.lld manually.

It would be nice to have OpenBSD support in before the next big refactor, I have time this weekend and going to give ocaml-freestanding on OpenBSD another crack.

Don't worry about the refactoring too much. Right now your work is mostly orthogonal to the immediate changes in the pipeline, I'll ping you if that changes and coordinate. Sorry for not yet getting around to helping with the ocaml-freestanding issues, I have my hands full with the refactoring.

Looks like i need to do some porting work and bring ocaml upto at least 4.04.2, which version would you recommend?

You may as well go with 4.06.0 if it's not too hard, the requirement is anything >= 4.04.2. I presume you have a working OPAM 2?

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Feb 14, 2018

I will go with 4.0.6.0, i have contacted @avsm as he is the maintainer for the port, but will get the changes done my self, if avsm has not already looked at it.

re OPAM2 umm, that is something i will also need to look into,

@mato

This comment has been minimized.

Copy link
Member

mato commented Feb 14, 2018

If you have OPAM 1.2 working then that's fine; you should be able to install that and then get 4.06.0 via OPAM. OPAM 2 is just a "nice to have" (faster, built-in solver, provides sane error messages when the dependencies fail).

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Feb 23, 2018

Finally got back to looking into this, but now something is wrong.

ocaml --version
The OCaml toplevel, version 4.06.0
opam --version
1.2.2

when i run the following

opam pin add ocaml-freestanding git://github.com/adamsteen/ocaml-freestanding

I get the following error

[NOTE] Package ocaml-freestanding is already git-pinned to git://github.com/adamsteen/ocaml-freestanding.
       This will erase any previous custom definition.
Proceed ? [Y/n] y


[ocaml-freestanding] git://github.com/adamsteen/ocaml-freestanding updated
[ocaml-freestanding] Installing new package description from git://github.com/adamsteen/ocaml-freestanding

ocaml-freestanding needs to be installed.
Your request can't be satisfied:
  - ocaml-freestanding.0.2.3 is not available because the package is pinned to git://github.com/adamsteen/ocaml-freestanding, version 0.2.3.

No solution found, exiting
[NOTE] Pinning command successful, but your installed packages may be out of sync.

Do i need to tweak my version to 0.2.3?

@mato

This comment has been minimized.

Copy link
Member

mato commented Feb 23, 2018

@mato

This comment has been minimized.

Copy link
Member

mato commented May 3, 2018

FYI, for the process of testing #226 and related MirageOS downstream PRs (of which all have been merged into at least the packages in Solo5/opam-solo5 now), here are the steps needed to get a working Solo5 + OPAM 2 installation on a fresh OpenBSD 6.3 install:

    # General dependencies
    doas pkg_add curl git gmake coreutils
    # To bootstrap OPAM 2 ("gmake cold" does not currently work)
    doas pkg_add ocaml
    # Default ulimit causes a stack overflow when building jbuilder
    ulimit -s 32768
    # Build and install OPAM 2
    git clone git://github.com/ocaml/opam && cd opam && ./configure && gmake lib-ext && gmake && \
    doas gmake install
    # OPAM 2 setup
    opam init
    # For opam-depext which it seems requires this
    doas pkg_add bash

After this proceed to create a switch with a recent compiler (I used 4.06.0) and add the Solo5/opam-solo5 remote, then do an opam pin solo5-kernel-ukvm to #226. You may need to do an opam pin edit ... to get the version number of the pin to match dev~solo5.

@mato

This comment has been minimized.

Copy link
Member

mato commented May 3, 2018

Note that until a new version of opam-depext with ocaml/opam-depext#95 is released you will also want to do opam pin add opam-depext git+https://github.com/ocaml/opam-depext.git#2.0 otherwise make depend on a Mirage unikernel will fail.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jun 1, 2018

@mato
with #226 being merged i am happy to close this! Unless you you would like me to chase down virtio and muen builds?

@mato

This comment has been minimized.

Copy link
Member

mato commented Jun 1, 2018

@adamsteen virtio and muen should build on OpenBSD, it'd be useful if you can confirm that.

Otherwise, I don't think it's worth porting solo5-run-virtio.sh to run on OpenBSD vmm, unless you need it yourself for some reason. Someone else can do it if they have a need.

What would be useful is to get an OpenBSD CI template; I have a base system template working in my nested-virt based infrastructure, but the main pain will be in getting surf-build to run. If you feel like dealing with node.js, you could try and get me a list of reproducible steps needed to get surf-build to run on OpenBSD-current. Ideally start with reverse-engineering/translating the steps in this Dockerfile: https://github.com/Solo5/solo5-ci/blob/master/any-Debian9-gcc630/Dockerfile.

If you decide to take this on, I'd suggest taking that to a separate, specific issue, probably on the Solo5-CI repository. So we can close this once I have some confirmation of the build status of virtio/muen from you.

@adamsteen

This comment has been minimized.

Copy link
Contributor

adamsteen commented Jun 15, 2018

@mato I have confirmed virtio and muen do build on OpenBSD, as per #261

Also I have opened issue OpenBSD CI to track building an OpenBSD CI template.

@mato

This comment has been minimized.

Copy link
Member

mato commented Jun 15, 2018

@adamsteen Thanks, closing this one then as we're all done here.

@mato mato closed this Jun 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment