New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building custom kernel modules fails in 1897.0.0 #2505

Closed
eranreshef opened this Issue Sep 20, 2018 · 10 comments

Comments

Projects
None yet
2 participants
@eranreshef

eranreshef commented Sep 20, 2018

Issue Report

Bug

Container Linux Version

1897.0.0

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1897.0.0
VERSION_ID=1897.0.0
BUILD_ID=2018-09-10-2308
PRETTY_NAME="Container Linux by CoreOS 1897.0.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

aws

Expected Behavior

I have an automated process for building a custom kernel module (wireguard) for coreos.
The process ran smoothly on previous versions (alpha, beta and stable).

Actual Behavior

When I executed the module's make command in 1897.0.0 I got:

ERROR: Kernel configuration is invalid.
include/generated/autoconf.h or include/config/auto.conf are missing.
Run 'make oldconfig && make prepare' on kernel src to fix it.

Reproduction Steps

  1. Follow the custom kernel module installation process - https://coreos.com/os/docs/latest/kernel-modules.html
  2. Follow wireguard's compilation process -
    https://www.wireguard.com/install/#compiling-from-source
@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Sep 20, 2018

Member

It's working for me. Can you paste your scripts?

I ran this in the developer container, and everything finished successfully:

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare
git clone --depth=1 https://git.zx2c4.com/WireGuard
make -C WireGuard/src
Member

dm0- commented Sep 20, 2018

It's working for me. Can you paste your scripts?

I ran this in the developer container, and everything finished successfully:

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare
git clone --depth=1 https://git.zx2c4.com/WireGuard
make -C WireGuard/src
@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Sep 22, 2018

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare

WIREGUARD_VERSION=0.0.20180918
filename=WireGuard-$WIREGUARD_VERSION
wget https://git.zx2c4.com/WireGuard/snapshot/$filename.tar.xz
tar xf $filename.tar.xz
make -C /$filename/src

Thanks for helping out.
I'm going on a short holiday and won't be available in the next couple of days, please don't hurry to close the issue if I don't respond.

eranreshef commented Sep 22, 2018

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare

WIREGUARD_VERSION=0.0.20180918
filename=WireGuard-$WIREGUARD_VERSION
wget https://git.zx2c4.com/WireGuard/snapshot/$filename.tar.xz
tar xf $filename.tar.xz
make -C /$filename/src

Thanks for helping out.
I'm going on a short holiday and won't be available in the next couple of days, please don't hurry to close the issue if I don't respond.

@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Sep 26, 2018

Member

I pasted all of those commands directly, and everything succeeded. Make sure your developer container version matches the host OS.

Member

dm0- commented Sep 26, 2018

I pasted all of those commands directly, and everything succeeded. Make sure your developer container version matches the host OS.

@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Oct 3, 2018

I download the dev container according to the instructions:

. /usr/share/coreos/release
. /usr/share/coreos/update.conf
url="http://${GROUP:-stable}.release.core-os.net/$COREOS_RELEASE_BOARD/$COREOS_RELEASE_VERSION/coreos_developer_container.bin.bz2"

eranreshef commented Oct 3, 2018

I download the dev container according to the instructions:

. /usr/share/coreos/release
. /usr/share/coreos/update.conf
url="http://${GROUP:-stable}.release.core-os.net/$COREOS_RELEASE_BOARD/$COREOS_RELEASE_VERSION/coreos_developer_container.bin.bz2"
@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Oct 4, 2018

This script is running on the coreOS machine:

#!/bin/sh
set -exuo pipefail

modules=/opt/modules
mkdir -p ${modules} ${modules}.wd
mount -o "lowerdir=/lib/modules,upperdir=${modules},workdir=${modules}.wd" -t overlay overlay /lib/modules
systemctl enable lib-modules.mount

. /usr/share/coreos/release
. /usr/share/coreos/update.conf
url="http://${GROUP:-stable}.release.core-os.net/$COREOS_RELEASE_BOARD/$COREOS_RELEASE_VERSION/coreos_developer_container.bin.bz2"
curl -L "$url" | bzip2 -d > coreos_developer_container.bin

# the coreos developer container is lacking of file system space since its downloading all the coreos sources to /usr/src
# we resolved the lack of space issue by creating an overlay on that dir
src_dir=$(pwd)/coreos_src
mkdir -p ${src_dir}
systemd-nspawn --bind=/lib/modules --bind=/tmp --overlay=${src_dir}::/usr/src --image=coreos_developer_container.bin /tmp/installer.sh

modprobe wireguard

and /tmp/installer.sh is:

#!/bin/bash
set -exuo pipefail
export WIREGUARD_VERSION=0.0.20180918

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare

# wireguard specific steps
emerge libmnl
filename=WireGuard-$WIREGUARD_VERSION
wget https://git.zx2c4.com/WireGuard/snapshot/$filename.tar.xz
tar xf $filename.tar.xz
make -C /$filename/src
make install -C /$filename/src

eranreshef commented Oct 4, 2018

This script is running on the coreOS machine:

#!/bin/sh
set -exuo pipefail

modules=/opt/modules
mkdir -p ${modules} ${modules}.wd
mount -o "lowerdir=/lib/modules,upperdir=${modules},workdir=${modules}.wd" -t overlay overlay /lib/modules
systemctl enable lib-modules.mount

. /usr/share/coreos/release
. /usr/share/coreos/update.conf
url="http://${GROUP:-stable}.release.core-os.net/$COREOS_RELEASE_BOARD/$COREOS_RELEASE_VERSION/coreos_developer_container.bin.bz2"
curl -L "$url" | bzip2 -d > coreos_developer_container.bin

# the coreos developer container is lacking of file system space since its downloading all the coreos sources to /usr/src
# we resolved the lack of space issue by creating an overlay on that dir
src_dir=$(pwd)/coreos_src
mkdir -p ${src_dir}
systemd-nspawn --bind=/lib/modules --bind=/tmp --overlay=${src_dir}::/usr/src --image=coreos_developer_container.bin /tmp/installer.sh

modprobe wireguard

and /tmp/installer.sh is:

#!/bin/bash
set -exuo pipefail
export WIREGUARD_VERSION=0.0.20180918

emerge-gitclone
emerge -gKv coreos-sources
gzip -cd /proc/config.gz > /usr/src/linux/.config
make -C /usr/src/linux modules_prepare

# wireguard specific steps
emerge libmnl
filename=WireGuard-$WIREGUARD_VERSION
wget https://git.zx2c4.com/WireGuard/snapshot/$filename.tar.xz
tar xf $filename.tar.xz
make -C /$filename/src
make install -C /$filename/src
@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Oct 17, 2018

ping... anyone there?

eranreshef commented Oct 17, 2018

ping... anyone there?

@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Oct 17, 2018

Member

Try adding KERNELDIR=/usr/src/linux to WireGuard's make commands.

Member

dm0- commented Oct 17, 2018

Try adding KERNELDIR=/usr/src/linux to WireGuard's make commands.

@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Oct 17, 2018

Wow! thanks! that did solve the issue!
Can you elaborate a bit more about the issue and the solution?
Is it a bug in my scripts?

eranreshef commented Oct 17, 2018

Wow! thanks! that did solve the issue!
Can you elaborate a bit more about the issue and the solution?
Is it a bug in my scripts?

@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Oct 18, 2018

Member

The reason you saw the problem and I didn't is because your script mounts /lib/modules from the host's production file system, and I didn't add that to the nspawn command when I tried.

Since an upgrade of the package manager a few weeks ago, it seems that they made a change in behavior that contradicts the documentation which resulted in kernel headers starting to be dropped from production images. They were still included in the developer container's file system, so our automated tests on it still passed.

I've added a workaround for that so it should work again in the next release.

Member

dm0- commented Oct 18, 2018

The reason you saw the problem and I didn't is because your script mounts /lib/modules from the host's production file system, and I didn't add that to the nspawn command when I tried.

Since an upgrade of the package manager a few weeks ago, it seems that they made a change in behavior that contradicts the documentation which resulted in kernel headers starting to be dropped from production images. They were still included in the developer container's file system, so our automated tests on it still passed.

I've added a workaround for that so it should work again in the next release.

@eranreshef

This comment has been minimized.

Show comment
Hide comment
@eranreshef

eranreshef Oct 18, 2018

Thanks for the explanation!

eranreshef commented Oct 18, 2018

Thanks for the explanation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment