Adapting modulus for general-purpose kernel module compilation

[PaulCapestany]

First off, just wanted to say thanks for this awesome repo and your KubeConEU presentation Lucas!

Quick side-question: at the 12:49 timestamp in the video of your talk, you briefly have a slide up showing the gdisk -l coreos_developer_container.bin command and mention that you'd come back to it if there was time. Looks like there wasn't time, or was the topic covered elsewhere?

Main question: (perhaps relatedly) it seemed like modulus might be a great way to automatically compile other (non-nvidia) modules for those of us running Kubernetes on CoreOS Container Linux, but, the 2.9G size of the chroot appears to end up being a bottleneck. I've been bumping against Fatal error: [...] No space left on device errors while playing around trying to adapt modulus to compile some in-tree kernel modules. It's unclear to me what the specific purpose of the truncate_bin() function is, and/or whether the dev container image base size could easily be made bigger in order to have more free space for compiling other modules? Would an approach along those lines make sense for folks interested in general-purpose automatic kernel module compilation, or is that barking up the wrong tree?

[squat]

@PaulCapestany thank you for the kind words!

Unfortunately, we ran out of time and weren't able to discuss that slide during the presentation. The discussion would have centered around the acrobatics needed to run developer container in another container. Specifically, the developer container is distributed as a raw disk image complete with GUID partition table and extra partitions that must either be skipped with an offset or be truncated. This image must then be mounted and chrooted.

With regards to your main question: yes, Modulus can absolutely be used to compile and install other kernel modules; in fact, it was deliberately designed this way. NVIDIA just happened to be the first one I needed and implemented. Modulus follows a plugin approach so that implementing compilation for any kernel module X means simply creating a directory modulus/X and providing two scripts: modulus/X/compile and modulus/X/install. Are you working on a particular kernel module? I'd love to give you a hand and add it to the project.

I too have run into issues with the developer container size particularly when trying to re-use the developer container. In order to circumvent this issue two things come to mind:

1. we can easily bump the size of the image (as you suggested); this works up til a certain point: we can try to make the file large enough that it is unlikely anyone will run into issues compiling kernel modules but we may need to be sensitive about using too much disk space.
2. we can mount the image, dump the contents of the image into a temporary directory, and then run the chroot there; this approach allows us to use as much space as we need without explicitly needing to allocate it.

What do you think?

For context, the truncate_bin function is used to remove the leading and trailing bits from the developer container so it can be mounted. Alternatively, we could pass the -o flag to the mount command and specify the offset.

[PaulCapestany]

With regards to your main question: yes, Modulus can absolutely be used to compile and install other kernel modules; in fact, it was deliberately designed this way.

@squat nice--I hoped/thought so but wanted to sanity-check; this is great to hear :)

Are you working on a particular kernel module? I'd love to give you a hand and add it to the project.

Yes, I have some toy-example-ish modules that I've been trying to get working (since afaict) being in-tree they seemed like they should be straightforward enough... I'm not sure if they'd directly be of interest to anyone else, however, getting them working would theoretically also show folks how to get other in-tree modules working?

Re: the particular module(s)--specifically, the ones I've been trying to get working have been Apple-hardware-specific modules such as apple_gmux, applesmc, etc. I've had a bare-metal CoreOS Container Linux HA k8s cluster set up via typhoon running for a while consisting of old Apple laptops, Mac Minis, Mac Pros, etc, which had previously been sitting unused/decommissioned, and it could be cool (but definitely not critical) to have more control/access over their hardware via the requisite kernel modules. Regardless, I'd think that sorting out a good way to get any/other in-tree module(s) working via modulus would probably prove useful to people and I'd be happy to try helping contribute to those ends either way.

I too have run into issues with the developer container size particularly when trying to re-use the developer container. In order to circumvent this issue two things come to mind:

1. we can easily bump the size of the image (as you suggested); this works up til a certain point: we can try to make the file large enough that it is unlikely anyone will run into issues compiling kernel modules but we may need to be sensitive about using too much disk space.
2. we can mount the image, dump the contents of the image into a temporary directory, and then run the chroot there; this approach allows us to use as much space as we need without explicitly needing to allocate it.

Approach 1. seemed like the obvious/easy route, but approach 2. sounds interesting if there might be situations where people would need somewhat unbounded space for compilations. Other than the observation that 1. seems superficially simpler, and that 2. might be more future-proof, I personally don't have very strong opinions in favor of one vs the other--which of them currently seems more compelling to you?

[PaulCapestany]

Ok, @squat thanks so much for clarifying the partition/truncation stuff, I think I got things working with a bigger base image. Just went with quick and dirty resize via truncate -s followed by resizing the filesystem while having dev image losetuped. This method could also be used to pass in an arg to allow for people to directly control size of volume they’d have

Currently AFK but I finally have my in-tree modules compiling away without running out of space, could post more later, thanks again for your help :)

[squat]

@PaulCapestany that’s great! Would you consider contributing the changes in a PR? It would be great to include the image resizing as well as any of the new kernel modules you are working on.
Thanks for the update.

[PaulCapestany]

Sure thing. Should be able to shoot off a PR later today/tomorrow

[PaulCapestany]

@squat I just opened PR #6 for the arbitrary-image-volume resizing feature we've been discussing here.

I'm still fiddling around with the general-purpose kernel module compilation stuff, but wanted to at least get this much out for now

[squat]

@PaulCapestany I'll close this for now since we don't have any blockers or issues.