New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kernel builder script #6

Open
jcvenegas opened this Issue Dec 5, 2017 · 11 comments

Comments

Projects
None yet
5 participants
@jcvenegas
Contributor

jcvenegas commented Dec 5, 2017

The script should manage:

  • config
  • fetch latest kernel from kata-containers/linux
@jcvenegas

This comment has been minimized.

Show comment
Hide comment
@jcvenegas

jcvenegas Dec 6, 2017

Contributor

@sameo @devimc @laijs, I'd like to automate the process to build kata kernel.

Based in comments from kata-containers/linux#5 we may want:

  • We want to have the configuration files in osbuilder tree.
  • The automation will pull the latest kata kernel
  • Will use the user desired configuration (kernel config)
  • Build a kernel !

What kernel formats we want to support ?

I consider vmlinux and vmlinuz (compressed and decompressed)

Do we want to support for kernel modules ?

IMO the kata kernel should not ship modules, all the need features would be built-in. This will make easy use the kernel independently of the guest image.

# Download the kernel
./kernel-builder fetch-kernel

Here I have in mind two ways:

  • Tag kata/linux repository and pull the latest tag from this repository (which have all the need patches for Kata kernel)

  • Have a file to save the latest kernel version we support

    • Download the kernel source from kernel.org or kernel repository and apply on top of it the config and the kernel patches for kata.

Then build the kernel:

# Download the kernel
./kernel-builder build-kernel

This will build the downloaded and configured kernel from ./kernel-builder fetch-kernel

Contributor

jcvenegas commented Dec 6, 2017

@sameo @devimc @laijs, I'd like to automate the process to build kata kernel.

Based in comments from kata-containers/linux#5 we may want:

  • We want to have the configuration files in osbuilder tree.
  • The automation will pull the latest kata kernel
  • Will use the user desired configuration (kernel config)
  • Build a kernel !

What kernel formats we want to support ?

I consider vmlinux and vmlinuz (compressed and decompressed)

Do we want to support for kernel modules ?

IMO the kata kernel should not ship modules, all the need features would be built-in. This will make easy use the kernel independently of the guest image.

# Download the kernel
./kernel-builder fetch-kernel

Here I have in mind two ways:

  • Tag kata/linux repository and pull the latest tag from this repository (which have all the need patches for Kata kernel)

  • Have a file to save the latest kernel version we support

    • Download the kernel source from kernel.org or kernel repository and apply on top of it the config and the kernel patches for kata.

Then build the kernel:

# Download the kernel
./kernel-builder build-kernel

This will build the downloaded and configured kernel from ./kernel-builder fetch-kernel

@bergwolf

This comment has been minimized.

Show comment
Hide comment
@bergwolf

bergwolf Dec 7, 2017

Member

Why should not kata kernel ship modules? Kernel features can be optional and we can keep optional modules separate without the need to load them at boot. It keeps the kernel size small as well as boosts kernel image loading.

Member

bergwolf commented Dec 7, 2017

Why should not kata kernel ship modules? Kernel features can be optional and we can keep optional modules separate without the need to load them at boot. It keeps the kernel size small as well as boosts kernel image loading.

@grahamwhaley

This comment has been minimized.

Show comment
Hide comment
@grahamwhaley

grahamwhaley Dec 7, 2017

Member

We discussed that somewhat already didn't we @bergwolf over at: kata-containers/linux#5 (comment)
So, it's not so much that we cannot enable modules, but we should make it a conscious decision on how we are going to handle storing or checking or enforcing the correct modules match up with the correct kernels - so do we enable module tagging and signing as well?

I know @mcastelino and I think @egernst would also like module support - and I have no real objection to enabling it as long as we have some plan/mitigation to help avert users from trying to load the wrong modules, which could then cause us many painful debug headaches :-)

Member

grahamwhaley commented Dec 7, 2017

We discussed that somewhat already didn't we @bergwolf over at: kata-containers/linux#5 (comment)
So, it's not so much that we cannot enable modules, but we should make it a conscious decision on how we are going to handle storing or checking or enforcing the correct modules match up with the correct kernels - so do we enable module tagging and signing as well?

I know @mcastelino and I think @egernst would also like module support - and I have no real objection to enabling it as long as we have some plan/mitigation to help avert users from trying to load the wrong modules, which could then cause us many painful debug headaches :-)

@bergwolf

This comment has been minimized.

Show comment
Hide comment
@bergwolf

bergwolf Dec 7, 2017

Member

@grahamwhaley yes we did and it seems we have not reached an agreement. We can enable module crc to have a minimal checking, or enable module signing if we care that much.

We should ship a list of modules we want to support and users are supposed to use them only. Users are responsible for their own actions inside a container and pay if they install a hacked module other than the ones provided by us. At the least, whatever modules loaded by a user only affect one guest.

Member

bergwolf commented Dec 7, 2017

@grahamwhaley yes we did and it seems we have not reached an agreement. We can enable module crc to have a minimal checking, or enable module signing if we care that much.

We should ship a list of modules we want to support and users are supposed to use them only. Users are responsible for their own actions inside a container and pay if they install a hacked module other than the ones provided by us. At the least, whatever modules loaded by a user only affect one guest.

@grahamwhaley

This comment has been minimized.

Show comment
Hide comment
@grahamwhaley

grahamwhaley Dec 7, 2017

Member

I think we can break this down into a couple of stages/decisions...

  1. I think if we enable module support we should also enable module versioning (CONFIG_MODVERSIONS=y) to try and prevent anybody loading a module that was not built for/against the actual kernel running. That is mostly to stop silly hard to debug errors :-). We should try to ensure that every kernel we build (at least with a different config file) has a different release strings to help enforce this (/cc @jcvenegas)
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/module-signing.rst#n281

  2. I agree @bergwolf that we should probably at least list the modules that we have tested that work :-)

  3. We should decide if we want to enable module signing as a security measure to stop anybody loading corrupted or hacked modules. I think we can delay that for a later day, and maybe make it a kernel/osbuilder build time decision for 'extra secure kata containers' etc.
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/module-signing.rst

Member

grahamwhaley commented Dec 7, 2017

I think we can break this down into a couple of stages/decisions...

  1. I think if we enable module support we should also enable module versioning (CONFIG_MODVERSIONS=y) to try and prevent anybody loading a module that was not built for/against the actual kernel running. That is mostly to stop silly hard to debug errors :-). We should try to ensure that every kernel we build (at least with a different config file) has a different release strings to help enforce this (/cc @jcvenegas)
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/module-signing.rst#n281

  2. I agree @bergwolf that we should probably at least list the modules that we have tested that work :-)

  3. We should decide if we want to enable module signing as a security measure to stop anybody loading corrupted or hacked modules. I think we can delay that for a later day, and maybe make it a kernel/osbuilder build time decision for 'extra secure kata containers' etc.
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/module-signing.rst

@bergwolf

This comment has been minimized.

Show comment
Hide comment
@bergwolf

bergwolf Dec 7, 2017

Member

@grahamwhaley Yes, I agree your all points. I just wanted to make sure we have module support and have a build/ship mechanism for modules ;) All the security considerations sound good to me.

Member

bergwolf commented Dec 7, 2017

@grahamwhaley Yes, I agree your all points. I just wanted to make sure we have module support and have a build/ship mechanism for modules ;) All the security considerations sound good to me.

@jcvenegas

This comment has been minimized.

Show comment
Hide comment
@jcvenegas

jcvenegas Dec 7, 2017

Contributor

@bergwolf @grahamwhaley thank you for the comments, I miss the discussion about modules in linux PR. I was afraid of having a dependency between kernel and image. All the points sounds fair.

Collecting information about it modules discussion:

I see two points:

  • Optional features are built as modules.
    crypto, nfs, vsock, ipv6, l2tp etc.
    They are not essential to run a pod and are not loaded if unneeded.
    Do we have a complete list of modules we want as module ( It would be great if they really increase a lot the kernel size and boot time) ?

  • Provide loadable modules opens up the opportunity to provide guest modules if the default kernel config misses something important to a user.

Contributor

jcvenegas commented Dec 7, 2017

@bergwolf @grahamwhaley thank you for the comments, I miss the discussion about modules in linux PR. I was afraid of having a dependency between kernel and image. All the points sounds fair.

Collecting information about it modules discussion:

I see two points:

  • Optional features are built as modules.
    crypto, nfs, vsock, ipv6, l2tp etc.
    They are not essential to run a pod and are not loaded if unneeded.
    Do we have a complete list of modules we want as module ( It would be great if they really increase a lot the kernel size and boot time) ?

  • Provide loadable modules opens up the opportunity to provide guest modules if the default kernel config misses something important to a user.

@grahamwhaley

This comment has been minimized.

Show comment
Hide comment
@grahamwhaley

grahamwhaley Dec 7, 2017

Member

For the list of modules to build by default, let's collate the needs from both CC and runv.
@egernst @mcastelino do you have some known needs for items you'd like to have available as modules?
@bergwolf , similarly, if you have items you know you want as modules can you list them here pls.

Then we can enable MODULES and MODVER in our default .config and add =m to the items on the list as a start. Then if/as we find more items are being requested regularly as modules we can update the default .config. We should also document how a user (can use osbuilder? to) build their own module to match the kernel if they need.

Member

grahamwhaley commented Dec 7, 2017

For the list of modules to build by default, let's collate the needs from both CC and runv.
@egernst @mcastelino do you have some known needs for items you'd like to have available as modules?
@bergwolf , similarly, if you have items you know you want as modules can you list them here pls.

Then we can enable MODULES and MODVER in our default .config and add =m to the items on the list as a start. Then if/as we find more items are being requested regularly as modules we can update the default .config. We should also document how a user (can use osbuilder? to) build their own module to match the kernel if they need.

@egernst

This comment has been minimized.

Show comment
Hide comment
@egernst

egernst Dec 7, 2017

Contributor

I know i40e offhand, but there may be others I'll need -- will follow up, likely on another issue.

Contributor

egernst commented Dec 7, 2017

I know i40e offhand, but there may be others I'll need -- will follow up, likely on another issue.

@jodh-intel

This comment has been minimized.

Show comment
Hide comment
@jodh-intel

jodh-intel Mar 21, 2018

Contributor

Hi @jcvenegas - I think we're going to need to look at this again as we move towards packaging Kata.

/cc @egernst.

Contributor

jodh-intel commented Mar 21, 2018

Hi @jcvenegas - I think we're going to need to look at this again as we move towards packaging Kata.

/cc @egernst.

@jodh-intel

This comment has been minimized.

Show comment
Hide comment
@jodh-intel

jodh-intel Jun 21, 2018

Contributor

Kernel configs

After the kernel config wiki page was created:

... I started looking at the kernel configs we currently have in the packaging repo:

... and I found the following (which I'm assuming was an oversight):

$ cd $GOPATH/src/github.com/kata-containers/packaging
$ grep "^CONFIG.*=m" kernel/configs/*4.14.x | cut -d: -f2- | sort -u | wc -l
11                              

Random thought

osbuilder creates a YAML file inside the image with details about the image. I'm wondering if we want to do something similar for the assets that live outside the image, one of which is the kernel. We could create something like:

  • /usr/share/kata-containers/vmlinux-${version}-container.yaml

    Containing:

    • full timestamp when file created.
    • the sha512 hash of the file (generated by osbuilder)
    • native architecture of the kernel.
    • a list of all the CONFIG_ options used to build the kernel for the current platform.
  • /usr/share/kata-containers/kata-containers-image-${version}.img.yaml

    Containing:

    • full timestamp when file created.
    • the sha512 hash of the file (generated by osbuilder)
    • native architecture of the image.
    • list of embedded kernel modules.

These files wouldn't need to be consumed by the runtime but would be useful for admins / developers.

What's annoying is that I can't see a way to check if a particular pair of files (kernel yaml + image yaml) are compatible. I think the best we could do is get osbuilder to generate a random hash and embed that into both YAML files so you'd atleast know that the pair were build as part of the same osbuilder run (and where tested together and hence should be compatible). Something like having the following in both files:

osbuilder-build-id: d0bccabbe133ad25f3bd0297b383baf58eef3c27
Contributor

jodh-intel commented Jun 21, 2018

Kernel configs

After the kernel config wiki page was created:

... I started looking at the kernel configs we currently have in the packaging repo:

... and I found the following (which I'm assuming was an oversight):

$ cd $GOPATH/src/github.com/kata-containers/packaging
$ grep "^CONFIG.*=m" kernel/configs/*4.14.x | cut -d: -f2- | sort -u | wc -l
11                              

Random thought

osbuilder creates a YAML file inside the image with details about the image. I'm wondering if we want to do something similar for the assets that live outside the image, one of which is the kernel. We could create something like:

  • /usr/share/kata-containers/vmlinux-${version}-container.yaml

    Containing:

    • full timestamp when file created.
    • the sha512 hash of the file (generated by osbuilder)
    • native architecture of the kernel.
    • a list of all the CONFIG_ options used to build the kernel for the current platform.
  • /usr/share/kata-containers/kata-containers-image-${version}.img.yaml

    Containing:

    • full timestamp when file created.
    • the sha512 hash of the file (generated by osbuilder)
    • native architecture of the image.
    • list of embedded kernel modules.

These files wouldn't need to be consumed by the runtime but would be useful for admins / developers.

What's annoying is that I can't see a way to check if a particular pair of files (kernel yaml + image yaml) are compatible. I think the best we could do is get osbuilder to generate a random hash and embed that into both YAML files so you'd atleast know that the pair were build as part of the same osbuilder run (and where tested together and hence should be compatible). Something like having the following in both files:

osbuilder-build-id: d0bccabbe133ad25f3bd0297b383baf58eef3c27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment