Integrate kernel bootstrapping from boot2now project

[rick-masters]

Please integration kernel bootstrapping from the boot2now project. Kernel bootstrapping works up to building Linux and has been integrated with live-bootstrap with a new --kernel-bootstrap option.

There is one last step remaining to complete the kernel bootstrap - a kexec of Linux from the Fiwix kernel. However, it may make sense to integrate with what works now in order to keep the integration up to date. Integration now would allow it to be tested and fixed if necessary when updates are made to live-bootstrap.

A PR request will be added to this issue. This PR is large and can be considered a first draft. I expect there will be substantial questions and suggestions which may lead to either updating the PR or creating a new one. I'd still like to improve the performance of builder-hex0 loading files at the beginning, improve the documentation, and do more testing.

The PR comes from the kernel-bootstrap-v1 branch of the http://github.com/rick-masters/live-bootstrap repo. So, testing of the integration can be done directly from that repo/branch.

[bittorf]

For me this stop here:

$ ./rootfs.py --kernel-bootstrap --qemu --update-checksums --external-sources

...
./tools/install.sh -D -m 644 include/syslog.h /tmp/destdir/usr/include/syslog.h
./tools/install.sh -D -m 644 include/tar.h /tmp/destdir/usr/include/tar.h
./tools/install.sh -D -m 644 include/termios.h /tmp/destdir/usr/include/termios.h
./tools/install.sh -D -m 644 include/tgmath.h /tmp/destdir/usr/include/tgmath.h
./tools/install.sh -D -m 644 include/threads.h /tmp/destdir/usr/include/threads.h
./tools/install.sh -D -m 644 include/time.h /tmp/destdir/usr/include/time.h
./tools/install.sh -D -m 644 include/uchar.h /tmp/destdir/usr/include/uchar.h
./tools/install.sh -D -m 644 include/ucontext.h /tmp/destdir/usr/include/ucontext.h
./tools/install.sh -D -m 644 include/ulimit.h /tmp/destdir/usr/include/ulimit.h
./tools/install.sh -D -m 644 include/unistd.h /tmp/destdir/usr/include/unistd.h
./tools/install.sh -D -m 644 include/utime.h /tmp/destdir/usr/include/utime.h
./tools/install.sh -D -m 644 include/utmp.h /tmp/destdir/usr/include/utmp.h
./tools/install.sh -D -m 644 include/utmpx.h /tmp/destdir/usr/include/utmpx.h
./tools/install.sh -D -m 644 include/values.h /tmp/destdir/usr/include/values.h
./tools/install.sh -D -m 644 include/wait.h /tmp/destdir/usr/include/wait.h
./tools/install.sh -D -m 644 include/wchar.h /tmp/destdir/usr/include/wchar.h
./tools/install.sh -D -m 644 include/wctype.h /tmp/destdir/usr/include/wctype.h
./tools/install.sh -D -m 644 include/wordexp.h /tmp/destdir/usr/include/wordexp.h
musl-1.2.3: creating package.
musl-1.2.3: cleaning up.
musl-1.2.3: installing package.
musl-1.2.3: build successful
Creating sysb rootfs
Creating sysb initramfs

WARNING: the last user process has exited. The kernel will stop itself.

[rick-masters]

@bittorf

Kernel bootstrapping works up to building Linux

This is as far as has been implemented. I'm glad it works for you :)

I still need to implement kexec from Fiwix to Linux to complete the implementation of kernel bootstrapping so this PR is just to start the process of integration.

[rick-masters]

I've run into difficulties switching Fiwix over to a disk drive. There are I/O errors and lock ups that I was not expecting. I also have not been putting as much time into development lately so resolving this has been delayed.

I'll probably be posting a new PR soon that is merged with latest live-bootstrap master and resolves a number of other issues which were raised in review but it would still be based on a ram drive. I want to see if it passes lint checks and so forth but it would still be a draft PR.

[fosslinux]

Thank you! Any and all progress updates is good, even if it hasn't addressed everything.

There are I/O errors and lock ups that I was not expecting

Are these errors Fiwix bugs, or yet undetermined? (If you have a WIP commit in a tree somewhere, I'd be happy to take a look at it when I get a chance (many eyes make all bugs shallow :P))

Also, force push to same PR (#253) is preferred instead of opening a brand new PR. Keeps all the discussion and code review in one place.

[rick-masters]

@fosslinux
I'm afraid my workflow has made updating the PR a lot more difficult than creating a new one. I'm using a rebase style of merge using updated patches rather than merging commits from master. I would have to do unnatural merge commits with git that I'd probably screw up. I probably should have set this up differently when I started. Sorry about that. On the bright side, a newer PR will be cleanly based on the current tip of the master branch and won't have any merge commits in it.

I'll be posting a new PR that is still based on a ram drive as was discussed in chat. I'm hopeful the disk issues can be worked out. I have a branch kernel-bootstrap-v2-fiwix that can be used to test Fiwix with a hard drive using the "--kernel-fiwix-hd" option (along with --qemu) and will be opening an Issue on the Fiwix repo so I can get help from Mikaku.

The new PR makes kernel bootstrapping the default. However, I included something a bit different. If the user copies a "kernel" file to the live-bootstrap directory it will continue to use that instead of kernel bootstrapping. However it outputs a warning that --kernel should be used and that implicit use of a kernel is deprecated. I thought this might avoid any surprises, especially since kernel bootstrapping is only partially implemented. I thought we could remove that behavior and require --kernel once I finished the kernel bootstrapping. I'm open to feedback on that however.

Sorry for the delay in providing this update. I won't be surprised if there are more lint issues or other issues that still need resolving but hopefully this is another good step forward.

[rick-masters]

I've been able to resolve the remaining lint errors. The CI failure revealed a complication. The CI runs under bwrap and it failed on the fiwix checksum. So the first problem is that bwrap shouldn't have been running the kernel bootstrap so I fixed that. But it raised the question of why the kernel bootstrap didn't catch the checksum problem. Well, the kernel bootstrap wasn't checking checksums because of a bug in the builder-hex0 kernel, which I have fixed. That revealed the fiwix checksum was wrong so I fixed that as well.
Unfortunately, I now see that the tcc-0.9.26 executables have different checksums when built on builder-hex0 vs Linux. So that's a new problem that I'm going to have to track down.