Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

building a rootfs without a kernel #493

Closed
tianon opened this issue Apr 10, 2024 · 10 comments
Closed

building a rootfs without a kernel #493

tianon opened this issue Apr 10, 2024 · 10 comments

Comments

@tianon
Copy link

tianon commented Apr 10, 2024

I've been using make root BUILTIN=1 to build just a rootfs (for use in containers, which I maintain builds of at https://hub.docker.com/r/tianon/toybox; easier to browse at https://oci.dag.dev/?repo=tianon/toybox). Because containers run directly on the host kernel, there's no need for this to contain a kernel (very similar to a chroot use case, really). 😄

As of the new 0.8.11 that is no longer working, with cryptic cpio errors. I think the refactoring in 97b776b is what made that stop working, but I'm not 100% sure. 🤔

Perhaps what I'm trying to do is actually an unsupported use case? I'm very open to suggestions here. 😄 🙇 ❤️

Output from the `Dockerfile` as-is (with the 0.8.11 bump):
Step 6/9 : RUN make root BUILTIN=1
 ---> Running in 62fc88de7ac7
mkroot/mkroot.sh  -- BUILTIN=1
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
	-DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/null Config.in > /dev/null
logpath:generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile /toybox/root/build/record-commands/logpath
...................
find: invalid argument 'f,l' to '-type'
find: invalid argument 'f,l' to '-type'
find: invalid argument 'f,l' to '-type'
find: invalid argument 'f,l' to '-type'
find: invalid argument 'f,l' to '-type'
find: invalid argument 'f,l' to '-type'
Building for host
mkroot/mkroot.sh: line 157: toybox: command not found
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== toybox
cleaned
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
	-DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/fd/63 Config.in > /dev/null
scripts/make.sh

warning: using unfinished code from toys/pending
generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile toybox
...............................................................................................................................................................................................................
scripts/install.sh --long --symlink --force
Compile instlist...
Install commands...
No $LINUX directory, kernel build skipped.
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== initramfs
mkroot/mkroot.sh: line 3[71](https://github.com/tianon/dockerfiles/actions/runs/8623651182/job/23637247602?pr=678#step:5:72): /toybox/root/host/docs/initramfs.cpio.gz: No such file or directory
find: unrecognized: -printf
cpio: unknown user/group +0:+0
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

	-L,-follow	Follow symlinks
	-H		...on command line only
	-xdev		Don't descend directories on other filesystems
	-maxdepth N	Descend at most N levels. -maxdepth 0 applies
			actions to command line arguments only
	-mindepth N	Don't act on first N levels
	-depth		Act on directory *after* traversing it

Actions:
	( ACTIONS )	Group actions for -o / -a
	! ACT		Invert ACT's success/failure
	ACT1 [-a] ACT2	If ACT1 fails, stop, else do ACT2
	ACT1 -o ACT2	If ACT1 succeeds, stop, else do ACT2
			Note: -a has higher priority than -o
	-name PATTERN	Match file name (w/o directory name) to PATTERN
	-iname PATTERN	Case insensitive -name
	-path PATTERN	Match path to PATTERN
	-ipath PATTERN	Case insensitive -path
	-regex PATTERN	Match path to regex PATTERN
	-type X		File type is X (one of: f,d,l,b,c,s,p)
	-executable	File is executable
	-perm MASK	At least one mask bit (+MASK), all bits (-MASK),
			or exactly MASK bits are set in file's mode
	-mtime DAYS	mtime is greater than (+N), less than (-N),
			or exactly N days in the past
	-atime DAYS	atime +N/-N/N days in the past
	-ctime DAYS	ctime +N/-N/N days in the past
	-mmin MINS	mtime is greater than (+N), less than (-N),
			or exactly N minutes in the past
	-newer FILE	mtime is more recent than FILE's
	-inum N		File has inode number N
	-user NAME/ID	File is owned by given user
	-group NAME/ID	File is owned by given group
	-size N[bck]	File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
			+/-N: file size is bigger/smaller than N
	-links N	Number of links is greater than (+N), less than (-N),
			or exactly N
	-empty		Match empty file/directory
	-prune		If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
	-print		Print file name
	-print0		Print file name, NUL terminated
	-exec CMD ARG ;	Run CMD with all instances of {} replaced by
			file name. Fails if CMD exits with nonzero
	-exec CMD ARG + Run CMD with {} replaced by list of file names
	-delete		Delete current file/directory. Turns on -depth option
	-quit		Exit
make: *** [Makefile:[81](https://github.com/tianon/dockerfiles/actions/runs/8623651182/job/23637247602?pr=678#step:5:82): root] Error 1
The command '/bin/sh -c make root BUILTIN=1' returned a non-zero code: 2
The same, but this time with GNU `find` and `cpio` (via Alpine's `findutils` and `cpio` packages) instead of the BusyBox implementations Alpine gives us by default to resolve some of the errors seen in the above output (but still ultimately failing):
Step 6/9 : RUN make root BUILTIN=1
 ---> Running in 2f0c8d3f2aa7
mkroot/mkroot.sh  -- BUILTIN=1
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
	-DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/null Config.in > /dev/null
logpath:generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile /toybox/root/build/record-commands/logpath
...................
find: ‘/usr/local/sbin’: No such file or directory
Building for host
mkroot/mkroot.sh: line 157: toybox: command not found
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== toybox
cleaned
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
	-DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/fd/63 Config.in > /dev/null
scripts/make.sh

warning: using unfinished code from toys/pending
generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile toybox
...............................................................................................................................................................................................................
scripts/install.sh --long --symlink --force
Compile instlist...
Install commands...
No $LINUX directory, kernel build skipped.
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== initramfs
mkroot/mkroot.sh: line 371: /toybox/root/host/docs/initramfs.cpio.gz: No such file or directory
cpio: blank line ignored
make: *** [Makefile:81: root] Error 1
The command '/bin/sh -c make root BUILTIN=1' returned a non-zero code: 2
@landley
Copy link
Owner

landley commented Apr 11, 2024

Huh, it works here. (You shouldn't need BUILTIN=1, that tells the kernel build to statically link the initramfs into the image rather than loading an external cpio.gz, shouldn't apply if you're not saying LINUX=/somewhere .)

The first error is indeed "busybox find -type f,l" not working, it doesn't know how to do fallbacks. Toybox added that in 2020 (commit 70a55cf).

This is a build under Alpine?

@landley
Copy link
Owner

landley commented Apr 11, 2024

I'm flying between cities today and unlikely to have a chance to set up an alpine VM, but I note that I recently added scripts/prereq/build.sh to create a minimal toybox capable of running the rest of the build. The initial commit was d1acc6e and I got it working on the mac (without homebrew) in commit 3bbc31c .

I need to do a proper fix so this "just works" again on alpine, but in the meantime you might be able to use the mac instructions to work around it?

@tianon
Copy link
Author

tianon commented Apr 12, 2024

I don't remember why I added BUILTIN=1; I think there was something that failed without it previously, but I'm certainly not attached to it! 😄

This is a build on Alpine, but I'm happy to make it not be! I'll swap it to Debian and/or try out your scripts/prereq/build.sh, and see if I can get a better result. 👍 👀

Also, definitely not urgent -- I hope your travels are safe! 🙇

@tianon
Copy link
Author

tianon commented Apr 12, 2024

Just pre-running scripts/prereq/build.sh makes it succeed! 💪

I think I will switch to a Debian-based builder image though, especially since even with that it gives me weird output from BusyBox and I don't really want to be chasing that all the time. 😄

Thanks so much for your help and for such a fun project. 👍

(docker run -it --rm tianon/toybox:0.8.11 coming soon! 🚀)

@tianon tianon closed this as completed Apr 12, 2024
@tianon
Copy link
Author

tianon commented Apr 12, 2024

Ahhh, I remember why I was using Alpine now -- because otherwise glibc gets grouchy about the static build and warns about functions like getgrouplist and friends being used. 🤦

I guess I'll go back to installing GNU tools in Alpine for now so the builds stay safe from glibc-isms and avoid the issues with BusyBox. 👍

@landley
Copy link
Owner

landley commented Apr 19, 2024

I tried to set up an alpine test environment (my last one was a chroot years ago), but it doesn't seem like they ship a livecd? Or at least the "extended" x86-64 image on their "downloads" page isn't one.

I downloaded their CD, kvm -m 2048 -cdrom blah.iso and got a login prompt instead of a desktop, the only account I could guess was "root", then I couldn't "git clone https://toybox" because it didn't have "git" installed. I googled and did an "apk add git" but it said it didn't know the package, "apk update" and "apk upgrade" didn't help...

This is not really a livecd.

@landley
Copy link
Owner

landley commented Apr 19, 2024

Used the setup program to install it to a virtual disk, booted that, logged in, installed git, logged in as the non-root user I'd created, cloned the repo, there was no make... and no sudo. And "apk add sudo" didn't work. Right... Ok, installed make, there was no gcc, installed that, and now it says ctype.h not found. I have to install an additional package to get standard posix headers supplied by musl, installing the compiler does not get me headers.

This is not the friendliest distro I've encountered. Also, what's the difference between the "extended" image and the "minimal" image?

Installed musl-dev. Installed bash. And now the build is complaining linux/rfkill.h isn't installed...

@landley
Copy link
Owner

landley commented Apr 20, 2024

I set up a busybox airlock directory (mv busybox dir; cd dir; for i in $(./busybox --list); do ln -s busybox $i; done) and stuck it at the start of the $PATH, and commit 7c6aecd build mkroot CROSS=i686 with that. (No LINUX= so it didn't build a kernel.)

Hopefully that means it works on alpine again. I replaced the -type x,y with ( -type x -o -type y ) because that's what busybox understands, and the airlock setup succeeding should handle the rest...

@tianon
Copy link
Author

tianon commented Apr 26, 2024

Oof, sorry you went through so much trouble on my behalf! Hopefully it felt productive for you? 🙈

If you need Alpine again (or want to set up some kind of CI / automated builds / testing) and are OK with a chroot, I'd suggest instead using the minirootfs tarballs from https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/ (or the suitable alternatives from another directory if you're on a different architecture ❤️). 😄


I tested using https://github.com/landley/toybox/archive/aedbaa5c4d451859b0e8b470e9520f722915bda0.tar.gz in place of the release tarballs (aedbaa5 being current HEAD), got rid of GNU's find and cpio so I was back to stock BusyBox tools, and got the following output and a successful end result:

Step 5/9 : RUN make root
 ---> Running in 15b164a9d71b
mkroot/mkroot.sh 
...................
find: /usr/local/sbin: No such file or directory
Building for host
mkroot/mkroot.sh: line 157: toybox: command not found
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== toybox
cleaned
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
	-DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/fd/63 Config.in > /dev/null
scripts/make.sh

warning: using unfinished code from toys/pending
generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile toybox
...............................................................................................................................................................................................................
scripts/install.sh --long --symlink --force
Compile instlist...
Install commands...
No $LINUX directory, kernel build skipped.
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== initramfs
find: unrecognized: -printf
cpio: unknown user/group +0:+0
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

	-L,-follow	Follow symlinks
	-H		...on command line only
	-xdev		Don't descend directories on other filesystems
	-maxdepth N	Descend at most N levels. -maxdepth 0 applies
			actions to command line arguments only
	-mindepth N	Don't act on first N levels
	-depth		Act on directory *after* traversing it

Actions:
	( ACTIONS )	Group actions for -o / -a
	! ACT		Invert ACT's success/failure
	ACT1 [-a] ACT2	If ACT1 fails, stop, else do ACT2
	ACT1 -o ACT2	If ACT1 succeeds, stop, else do ACT2
			Note: -a has higher priority than -o
	-name PATTERN	Match file name (w/o directory name) to PATTERN
	-iname PATTERN	Case insensitive -name
	-path PATTERN	Match path to PATTERN
	-ipath PATTERN	Case insensitive -path
	-regex PATTERN	Match path to regex PATTERN
	-type X		File type is X (one of: f,d,l,b,c,s,p)
	-executable	File is executable
	-perm MASK	At least one mask bit (+MASK), all bits (-MASK),
			or exactly MASK bits are set in file's mode
	-mtime DAYS	mtime is greater than (+N), less than (-N),
			or exactly N days in the past
	-atime DAYS	atime +N/-N/N days in the past
	-ctime DAYS	ctime +N/-N/N days in the past
	-mmin MINS	mtime is greater than (+N), less than (-N),
			or exactly N minutes in the past
	-newer FILE	mtime is more recent than FILE's
	-inum N		File has inode number N
	-user NAME/ID	File is owned by given user
	-group NAME/ID	File is owned by given group
	-size N[bck]	File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
			+/-N: file size is bigger/smaller than N
	-links N	Number of links is greater than (+N), less than (-N),
			or exactly N
	-empty		Match empty file/directory
	-prune		If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
	-print		Print file name
	-print0		Print file name, NUL terminated
	-exec CMD ARG ;	Run CMD with all instances of {} replaced by
			file name. Fails if CMD exits with nonzero
	-exec CMD ARG + Run CMD with {} replaced by list of file names
	-delete		Delete current file/directory. Turns on -depth option
	-quit		Exit
Output is in /toybox/root/host
Removing intermediate container 15b164a9d71b

(happy to share more details if you want them, or test more things, but this seems good 👍)


Again, I really appreciate it, and enjoy playing with and using Toybox. 😁

@landley
Copy link
Owner

landley commented Apr 27, 2024

Ah, the downloadable chroot is probably what I used last time.

I want it to work, you brought me an example of it not working, I'm gnawing at it.

Huh, it looks like it's not installing the airlock (which is why attempting to set VERSION via toybox --version is not finding it). I wonder why?

The /dev/tty thing is harmless (announce is trying to change your title bar), I suppose I could add a 2>/dev/null to suppress it, I've just never encountered it before...

The busybox find -printf thing at the end is again, because busybox doesn't support an option both toybox and gnu do, and it didn't build the airlock for some reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants