This repository has been archived by the owner. It is now read-only.

mkinitfs: terrible hack to mount rootfs over NFS (WIP) #547

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@zhuowei
Collaborator

zhuowei commented Sep 10, 2017

Very hacky proof of concept for #511: this mounts /boot and / on Nexus 6P (angler) over NFS over USB.

Do not merge this.

Changes:

  • Removed existing root/boot detection code
  • Changed mount commands to mount as NFS
  • Changed DHCP range to only give out one IP address.

I'm serving NFS using my host system (Ubuntu 14.04)'s nfs-kernel-server, and I've rebuilt my device's kernel to add NFS support. The scripts I used on the host system to start NFS is here.

The way it's currently implemented prevents mounting the root FS normally. What would be a good way to integrate this into the normal initfs? Also, can we serve NFS from inside the Alpine chroot or do we need to serve NFS from the host system?

@PabloCastellano

This comment has been minimized.

Show comment
Hide comment
@PabloCastellano

PabloCastellano Sep 10, 2017

Member

I've changed the title and added dont_merge_pr label instead

Member

PabloCastellano commented Sep 10, 2017

I've changed the title and added dont_merge_pr label instead

@PabloCastellano PabloCastellano changed the title from WIP: Do NOT merge: mkinitfs: terrible hack to mount rootfs over NFS to mkinitfs: terrible hack to mount rootfs over NFS (WIP) Sep 10, 2017

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Sep 10, 2017

Member

The way it's currently implemented prevents mounting the root FS normally. What would be a good way to integrate this into the normal initfs?

I would check the kernel parameters, like we do for disabling the log redirection already.

Also, can we serve NFS from inside the Alpine chroot or do we need to serve NFS from the host system?

It would be in the spirit of the current pmbootstrap code, to automatize everything as much as possible and therefore to set it all up from inside the Alpine chroot with pmbootstrap. From a user's point of view, I would like to have it as simple as possible. One example I can think of right now: pmbootstrap flasher boot --nfs and it sets up the NFS server inside the chroot, and restricts it so only the phone can access it. pmbootstrap shutdown should stop the NFS server again (like we stop the distccd and abd servers running inside the native chroot already).

During the installation, we already mount the device chroot in the native chroot, so you will probably also need to do that, in order to have the nfs server running inside the native chroot.

Please note that I did not make a code review yet, please tell me when you think it is ready to be reviewed or if you would like to have one right now anyway.

Thank you for working on this, I'm really looking forward to this feature. With this, we could really develop postmarketOS on devices without changing them at all, even if they do not have an sdcard slot!

Member

ollieparanoid commented Sep 10, 2017

The way it's currently implemented prevents mounting the root FS normally. What would be a good way to integrate this into the normal initfs?

I would check the kernel parameters, like we do for disabling the log redirection already.

Also, can we serve NFS from inside the Alpine chroot or do we need to serve NFS from the host system?

It would be in the spirit of the current pmbootstrap code, to automatize everything as much as possible and therefore to set it all up from inside the Alpine chroot with pmbootstrap. From a user's point of view, I would like to have it as simple as possible. One example I can think of right now: pmbootstrap flasher boot --nfs and it sets up the NFS server inside the chroot, and restricts it so only the phone can access it. pmbootstrap shutdown should stop the NFS server again (like we stop the distccd and abd servers running inside the native chroot already).

During the installation, we already mount the device chroot in the native chroot, so you will probably also need to do that, in order to have the nfs server running inside the native chroot.

Please note that I did not make a code review yet, please tell me when you think it is ready to be reviewed or if you would like to have one right now anyway.

Thank you for working on this, I'm really looking forward to this feature. With this, we could really develop postmarketOS on devices without changing them at all, even if they do not have an sdcard slot!

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Sep 10, 2017

Member

I think we need to modify the DHCP server on the phone so it only gives out a single IP address to the host machine (like 172.16.42.2) and then start the NFS kernel server and bind it to that IP address.

I think we could implement it neater in the initfs by checking a kernel parameter for nfs booting. like passing pmos.nfsboot=true to the kernel cmdline.

Member

MartijnBraam commented Sep 10, 2017

I think we need to modify the DHCP server on the phone so it only gives out a single IP address to the host machine (like 172.16.42.2) and then start the NFS kernel server and bind it to that IP address.

I think we could implement it neater in the initfs by checking a kernel parameter for nfs booting. like passing pmos.nfsboot=true to the kernel cmdline.

@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Sep 10, 2017

Collaborator

@MartijnBraam This PR does limit the DHCP IP to a single address; in an actual implementation one would switch the range based on whether NFS is enabled on the kernel command line.

@ollieparanoid For NFS in a Chroot, one needs:

  • the nfsd kernel module to be modprobe'd outside the chroot
  • two processes running inside the chroot: rpc.mount and rpc.nfsd
  • the host system's nfsd to be disabled

The first can be done using similar code to the losetup code we already have. The second I'm a bit iffy about, since distccd only uses one process, while this one needs two. We would probably detect the host nfsd somehow and tell the user to disable it.

Also, should we NFS mount the image directly, or do we need to make a copy of the system root partition prior to mounting?

Finally, this does require NFS client to be enabled in the device's kernel. Should we use a different protocol instead (e.g. a FUSE-based FS) so that we don't have to change the kernel config?

Collaborator

zhuowei commented Sep 10, 2017

@MartijnBraam This PR does limit the DHCP IP to a single address; in an actual implementation one would switch the range based on whether NFS is enabled on the kernel command line.

@ollieparanoid For NFS in a Chroot, one needs:

  • the nfsd kernel module to be modprobe'd outside the chroot
  • two processes running inside the chroot: rpc.mount and rpc.nfsd
  • the host system's nfsd to be disabled

The first can be done using similar code to the losetup code we already have. The second I'm a bit iffy about, since distccd only uses one process, while this one needs two. We would probably detect the host nfsd somehow and tell the user to disable it.

Also, should we NFS mount the image directly, or do we need to make a copy of the system root partition prior to mounting?

Finally, this does require NFS client to be enabled in the device's kernel. Should we use a different protocol instead (e.g. a FUSE-based FS) so that we don't have to change the kernel config?

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Sep 10, 2017

Member

An alternative to nfs might be booting from nbd (which I can't find a lot of information about). That way we boot the system.img directly over the network.

Member

MartijnBraam commented Sep 10, 2017

An alternative to nfs might be booting from nbd (which I can't find a lot of information about). That way we boot the system.img directly over the network.

@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Sep 10, 2017

Collaborator

@MartijnBraam NBD also needs kernel support on the device, right?

Collaborator

zhuowei commented Sep 10, 2017

@MartijnBraam NBD also needs kernel support on the device, right?

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Sep 10, 2017

Member

Yep but luckily we control the kernel on the device so we can add NFS or NBD

Member

MartijnBraam commented Sep 10, 2017

Yep but luckily we control the kernel on the device so we can add NFS or NBD

nfs-root: mount a /tmp ramdisk
The default root image doesn't have enough space for Weston to create files
in /tmp. Mount a ramdisk there instead.
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Sep 11, 2017

Member

Also, should we NFS mount the image directly, or do we need to make a copy of the system root partition prior to mounting?

The generated system image is just the device chroot copied into an image file. Unless we get performance problems, I would prefer to directly use the device chroot if possible.

That way we would not have any size limitations (the system image is just as big as the chroot and gets resized when copied to the device, see the qemu wiki article on how to resize it) and the test cycle would be much faster:

% pmbootstrap chroot -r
# / apk add gimp
% pmbootstrap flasher boot --nfs

This would be all, that is necessary to install a program - no need to rebuild the whole image and loose all changes you may have made. (And if you want to rebuild it, simply pmbootstrap zap.)

With that being said, I think a blockdevice is probably faster, which would speak for nbd.

How about implementing the nfs boot with directly accessing the rootfs folder like I've suggested above first, and when that does not perform, look into nbd (maybe offer both options)?

Member

ollieparanoid commented Sep 11, 2017

Also, should we NFS mount the image directly, or do we need to make a copy of the system root partition prior to mounting?

The generated system image is just the device chroot copied into an image file. Unless we get performance problems, I would prefer to directly use the device chroot if possible.

That way we would not have any size limitations (the system image is just as big as the chroot and gets resized when copied to the device, see the qemu wiki article on how to resize it) and the test cycle would be much faster:

% pmbootstrap chroot -r
# / apk add gimp
% pmbootstrap flasher boot --nfs

This would be all, that is necessary to install a program - no need to rebuild the whole image and loose all changes you may have made. (And if you want to rebuild it, simply pmbootstrap zap.)

With that being said, I think a blockdevice is probably faster, which would speak for nbd.

How about implementing the nfs boot with directly accessing the rootfs folder like I've suggested above first, and when that does not perform, look into nbd (maybe offer both options)?

@drebrez

This comment has been minimized.

Show comment
Hide comment
@drebrez

drebrez Jan 18, 2018

Member

What about implementing this feature as an initramfs hook?
The user will have to do pmbootstrap initfs hook_add nfs_boot and he will be informed about the 2 nfs share that have to expose.

Member

drebrez commented Jan 18, 2018

What about implementing this feature as an initramfs hook?
The user will have to do pmbootstrap initfs hook_add nfs_boot and he will be informed about the 2 nfs share that have to expose.

@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Jan 18, 2018

Collaborator

@drebrez I don't think initramfs hooks can override the rootfs mounting: even if a hook mounts root, the startup script would still complain that it can't find the root fs.

Collaborator

zhuowei commented Jan 18, 2018

@drebrez I don't think initramfs hooks can override the rootfs mounting: even if a hook mounts root, the startup script would still complain that it can't find the root fs.

@drebrez

This comment has been minimized.

Show comment
Hide comment
@drebrez

drebrez Jan 18, 2018

Member

@zhuowei because you consider leaving the hook script and continue with the normal init script. If the hook itself handles everything (mount nfs partitions and exec switch_root), then it should be fine, no?

Member

drebrez commented Jan 18, 2018

@zhuowei because you consider leaving the hook script and continue with the normal init script. If the hook itself handles everything (mount nfs partitions and exec switch_root), then it should be fine, no?

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jan 18, 2018

Member

I think implementing it as a hook is a good idea! 👍

Member

ollieparanoid commented Jan 18, 2018

I think implementing it as a hook is a good idea! 👍

@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Feb 23, 2018

Collaborator

@ollieparanoid I would appreciate some guidance on getting this to a working state. I originally tested this by manually exporting the NFS directories from my host operating system. I would like to automate that, but I have a few questions.

My ideal user interface for this would be an extra --nfs param added to flasher boot, which would automatically start an NFS server and configure the kernel cmdline to look for it. Is this possible, or desirable?

Is it possible to even run an NFS server in a chroot? (I can get the server daemon to start, but I have not tested actually exporting any filesystems).

What if the user already has an NFS server running outside the chroot? I don't think one cannot run two NFS servers at the same time, since a special /proc/fs/nfsd filesystem must be mounted to start a server.

NFS is not one daemon but several (rpcbind, rpc.nfsd, and (for NFSv3) rpc.mountd). How would we check all their statuses/shut them all down at once?

Should we use NFSv3 (the most common protocol), NFSv4 (newest version), or use something other than NFS, such as NBD?

Collaborator

zhuowei commented Feb 23, 2018

@ollieparanoid I would appreciate some guidance on getting this to a working state. I originally tested this by manually exporting the NFS directories from my host operating system. I would like to automate that, but I have a few questions.

My ideal user interface for this would be an extra --nfs param added to flasher boot, which would automatically start an NFS server and configure the kernel cmdline to look for it. Is this possible, or desirable?

Is it possible to even run an NFS server in a chroot? (I can get the server daemon to start, but I have not tested actually exporting any filesystems).

What if the user already has an NFS server running outside the chroot? I don't think one cannot run two NFS servers at the same time, since a special /proc/fs/nfsd filesystem must be mounted to start a server.

NFS is not one daemon but several (rpcbind, rpc.nfsd, and (for NFSv3) rpc.mountd). How would we check all their statuses/shut them all down at once?

Should we use NFSv3 (the most common protocol), NFSv4 (newest version), or use something other than NFS, such as NBD?

@MartijnBraam

This comment has been minimized.

Show comment
Hide comment
@MartijnBraam

MartijnBraam Feb 23, 2018

Member

NFS is kinda ugly to deal with in a chroot, I think you might need a seperate network namespace to run nfs on the host and a chroot at the same time (and thus making the chroot dangerously close to a container). It might be neater to do the nfs server on the host (since you need the stuff on the host anyway to make it work in the chroot so this doesn't really add another dependency)

Member

MartijnBraam commented Feb 23, 2018

NFS is kinda ugly to deal with in a chroot, I think you might need a seperate network namespace to run nfs on the host and a chroot at the same time (and thus making the chroot dangerously close to a container). It might be neater to do the nfs server on the host (since you need the stuff on the host anyway to make it work in the chroot so this doesn't really add another dependency)

@zhuowei

This comment has been minimized.

Show comment
Hide comment
@zhuowei

zhuowei Feb 23, 2018

Collaborator

@MartijnBraam Most Linux distros install the NFS kernel module by default but not the actual NFS server, so pmbootstrap would have to install that through the host's package manager. That might be annoying to setup for multiple distros.

As for NFS conflicts: can I just assume that no-one is serving NFS from the host system? It's not exactly a commonly run service...

Collaborator

zhuowei commented Feb 23, 2018

@MartijnBraam Most Linux distros install the NFS kernel module by default but not the actual NFS server, so pmbootstrap would have to install that through the host's package manager. That might be annoying to setup for multiple distros.

As for NFS conflicts: can I just assume that no-one is serving NFS from the host system? It's not exactly a commonly run service...

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Feb 24, 2018

Member

My ideal user interface for this would be an extra --nfs param added to flasher boot, which would automatically start an NFS server and configure the kernel cmdline to look for it. Is this possible, or desirable?

Yes, I think that would make it easy to use and pbmootstrap flasher boot --nfs sounds intuitive to me (actually I've suggested the same a few posts upwards). I would prefer if we do it in the chroot and not in the host system, so it works without touching the host system and we can use Alpine versions everywhere, not something host OS specific.

Is it possible to even run an NFS server in a chroot? (I can get the server daemon to start, but I have not tested actually exporting any filesystems).

I couldn't find a source that says: Yes it is possible, so we need to try that out. But I am confident that it is possible one way or the other. What I found:

What if the user already has an NFS server running outside the chroot? I don't think one cannot run two NFS servers at the same time, since a special /proc/fs/nfsd filesystem must be mounted to start a server.

As for NFS conflicts: can I just assume that no-one is serving NFS from the host system? It's not exactly a commonly run service...

I think we can assume that the host system doesn't run NFS. With that being said, it should be possible to run the user-mode implementation side by side with the host system daemon.

NFS is not one daemon but several (rpcbind, rpc.nfsd, and (for NFSv3) rpc.mountd). How would we check all their statuses/shut them all down at once?

pmb/chroot/distccd.py is a good example of how we could handle this. Basically we save the PIDs of all the daemons in some temporary file, and use it to check if the daemon is still running when it is needed. And to shut them down with kill.

When introducing the NFS daemon(s), I think it makes sense to refactor some functions from that file and make them usable for daemons in the chroot in general (get_running_pid, is_running, start, stop). We could put them in a new file pmb/chroot/daemon.py and call them from distccd.py and a new nfsd.py.

(NOTE: We also have the adb daemon running in the chroot when executing pmbootstrap flasher --method=adb sideload, but it did not really work with PIDs, so the logic for that is different.)

Should we use NFSv3 (the most common protocol), NFSv4 (newest version), or use something other than NFS, such as NBD?

From reading up on this, NFSv4 seems to be simpler, and it is around for quite a long time now, so we should be fine with NFSv4 only (for the official server, when using the alternative server linked above we should get additional NVSv3 without extra effort as I understand).

I prefer NFS over NDB, because with NFS we could implement something like:

$ pmbootstrap install --no-image
$ pmbootstrap boot --nfs

and you could seamlessly access the chroot and make changes from your PC with:

$ pmbootstrap chroot -r

In contrary with NDB we would need to generate the image file and resize it appropriately, which takes up more space and more time (to copy everything to the image). And to access the image, you would have to mount it, and there's no easy way to chroot into it with pmbootstrap.

$ pmbootstrap install
$ pmbootstrap boot --ndb --image-size=2G
Member

ollieparanoid commented Feb 24, 2018

My ideal user interface for this would be an extra --nfs param added to flasher boot, which would automatically start an NFS server and configure the kernel cmdline to look for it. Is this possible, or desirable?

Yes, I think that would make it easy to use and pbmootstrap flasher boot --nfs sounds intuitive to me (actually I've suggested the same a few posts upwards). I would prefer if we do it in the chroot and not in the host system, so it works without touching the host system and we can use Alpine versions everywhere, not something host OS specific.

Is it possible to even run an NFS server in a chroot? (I can get the server daemon to start, but I have not tested actually exporting any filesystems).

I couldn't find a source that says: Yes it is possible, so we need to try that out. But I am confident that it is possible one way or the other. What I found:

What if the user already has an NFS server running outside the chroot? I don't think one cannot run two NFS servers at the same time, since a special /proc/fs/nfsd filesystem must be mounted to start a server.

As for NFS conflicts: can I just assume that no-one is serving NFS from the host system? It's not exactly a commonly run service...

I think we can assume that the host system doesn't run NFS. With that being said, it should be possible to run the user-mode implementation side by side with the host system daemon.

NFS is not one daemon but several (rpcbind, rpc.nfsd, and (for NFSv3) rpc.mountd). How would we check all their statuses/shut them all down at once?

pmb/chroot/distccd.py is a good example of how we could handle this. Basically we save the PIDs of all the daemons in some temporary file, and use it to check if the daemon is still running when it is needed. And to shut them down with kill.

When introducing the NFS daemon(s), I think it makes sense to refactor some functions from that file and make them usable for daemons in the chroot in general (get_running_pid, is_running, start, stop). We could put them in a new file pmb/chroot/daemon.py and call them from distccd.py and a new nfsd.py.

(NOTE: We also have the adb daemon running in the chroot when executing pmbootstrap flasher --method=adb sideload, but it did not really work with PIDs, so the logic for that is different.)

Should we use NFSv3 (the most common protocol), NFSv4 (newest version), or use something other than NFS, such as NBD?

From reading up on this, NFSv4 seems to be simpler, and it is around for quite a long time now, so we should be fine with NFSv4 only (for the official server, when using the alternative server linked above we should get additional NVSv3 without extra effort as I understand).

I prefer NFS over NDB, because with NFS we could implement something like:

$ pmbootstrap install --no-image
$ pmbootstrap boot --nfs

and you could seamlessly access the chroot and make changes from your PC with:

$ pmbootstrap chroot -r

In contrary with NDB we would need to generate the image file and resize it appropriately, which takes up more space and more time (to copy everything to the image). And to access the image, you would have to mount it, and there's no easy way to chroot into it with pmbootstrap.

$ pmbootstrap install
$ pmbootstrap boot --ndb --image-size=2G
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Feb 24, 2018

Member

since you need the stuff on the host anyway to make it work in the chroot

If we really need to run something on the host for the official NFS implementation, then I'd go with the user-mode implementation which should run without problems in the chroot.

Member

ollieparanoid commented Feb 24, 2018

since you need the stuff on the host anyway to make it work in the chroot

If we really need to run something on the host for the official NFS implementation, then I'd go with the user-mode implementation which should run without problems in the chroot.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.