Alpine 3.12 #72

mmlb · 2020-07-17T21:38:52Z

Description

Update our x86_64 alpine to v3.12, upgrade kernel to latest lts (5.4.52) and add Pensando NIC drivers. Also lift the rules from installer/alpine/Makefile into rules.mk.j2 so that the top-level make can handle building the alpine files itself.

Why is this needed

Fixes: #1

How Has This Been Tested?

Tested via the qemu based tests, also in production on various hardware.

How are existing users impacted? What migration steps/scripts do we need?

Newer kernel means better hardware support. A supported Alpine base means bug fixes, especially ca-certificates updates :D.

mikemrm · 2020-07-17T23:24:42Z

installer/alpine/Dockerfile

+    # remove edge when linux-vanilla is > 4.14.8 and has the nfp driver enabled,
+    # likely alpine v3.8
+    apk add --no-scripts --no-cache --update --upgrade --repository http://dl-cdn.alpinelinux.org/alpine/edge/main \


Can we see if this is no longer needed? I believe this was to get around some old packages.

the comment doesn't make any more sense but we still want latest and greatest linux-firmare. I tried building off of the pinned aports config but it wasn't working for me yesterday. Checked again today and it looks like the dev.alpinelinux.org host is back up again. Will use that then.

Now installing linux-firmware from the checked out aports

mikemrm · 2020-07-17T23:36:04Z

rules.mk.j2

 build/osie-aarch64.tar.gz: SED=/FROM/ s|.*|FROM multiarch/ubuntu-debootstrap:arm64-xenial|
-build/osie-x86_64.tar.gz: SED=
+build/osie-x86_64.tar.gz:  SED=


extra space?

was on purpose for alignment ;p. Doesn't hurt.

mikemrm · 2020-07-17T23:36:12Z

rules.mk.j2

 build/osie-runner-aarch64.tar.gz: SED=/FROM/ s|.*|FROM multiarch/alpine:arm64-v3.7|
-build/osie-runner-x86_64.tar.gz: SED=
+build/osie-runner-x86_64.tar.gz:  SED=


mikemrm · 2020-07-17T23:38:11Z

rules.mk.j2

 build/osie-%.tar.gz: docker/Dockerfile ${osiesrcs}
 	$(E)"DOCKER   $@"
 	$(Q)sed '${SED}' $< > $<.$*
 	$(Q)docker build -t osie:$* -f $<.$* $(<D) 2>&1 | tee $@.log >/dev/$T
 	$(Q)docker save osie:$* > $@.tmp
 	$(Q)mv $@.tmp $@

+build/osie-runner-aarch64.tar.gz: /proc/sys/fs/binfmt_misc/qemu-aarch64
 build/osie-runner-aarch64.tar.gz: SED=/FROM/ s|.*|FROM multiarch/alpine:arm64-v3.7|


Do you want to wrap these in an if statement like line 180?

nope because this is hard coded to aarch64, and so we don't need the if.

## Description Lift the rules from installer/alpine/Makefile into rules.mk.j2 so that the top-level make can handle building the alpine files itself. Also cleanup/fix the Dockerfile & build.sh files that are responsible for building the Alpine boot files. ## Why is this needed Extracted out of #72 for better reviewability. ## How Has This Been Tested? Builds kernel, initramfs, and modloop files as needed.

Needed to upgrade deps since `six` is a disutils package and pip was having trouble with it. I manually held `grpcio` back since it fails to build in Alpine 3.12 [1]. [1]: grpc/grpc#23742 Signed-off-by: Manuel Mendez <manny@packet.com>

Signed-off-by: Manuel Mendez <manny@packet.com>

Going to need a newer kernel for the Pensando NICs and might as well use the latest LTS release. Signed-off-by: Manuel Mendez <manny@packet.com>

Its not great for reproducibility to install from edge at any random point in time of the repo. Since we already have a checkout of aports, might as well use that. Signed-off-by: Manuel Mendez <manny@packet.com>

Similar to linux-firmware its not great for reproducibility to install from edge at any random point in time of the repo. Since we already have a checkout of aports, might as well use that. Signed-off-by: Manuel Mendez <manny@packet.com>

Signed-off-by: Manuel Mendez <manny@packet.com>

mikemrm

lgtm!

For better caching/re-runs. There is no point in agonizing over docker layers as we do not ship this image anywhere. Signed-off-by: Manuel Mendez <manny@packet.com>

We don't really need to install kexec-tools, was being done so we can have the package cached so its embedded in the initramfs. The packages we do need installed are done at the end so we can minimize keeping track of the cache dir. Signed-off-by: Manuel Mendez <manny@packet.com>

Makes sense to build kexec-tools after the kernel has been built and installed instead of the other way around. This will also be needed to make using the build output as a local repo. Signed-off-by: Manuel Mendez <manny@packet.com>

I could not figure out how to make it so apk will recognize the kexec-tools apk files in the cache dir as valid. Luckily abuild sets up a local repo and we can copy it into the initramfs and setup a new local repo for it. Signed-off-by: Manuel Mendez <manny@packet.com>

mkintfs is installed from v3.4.5 in the Dockefile, which maps to https://gitlab.alpinelinux.org/alpine/mkinitfs/-/raw/v3.4.5/initramfs-init.in Signed-off-by: Manuel Mendez <manny@packet.com>

@Version

Fetched from https://raw.githubusercontent.com/alpinelinux/mkinitfs/v3.4.5/initramfs-init.in and replaced @Version@ with tag as is done in the Makefile. Signed-off-by: Manuel Mendez <manny@packet.com>

Makes a bunch of tools happier, shfmt, vim syntax highlighting. Signed-off-by: Manuel Mendez <manny@packet.com>

Avoids spraying console with uninteresting progress output. Signed-off-by: Manuel Mendez <manny@packet.com>

Otherwise docker won't start up. Signed-off-by: Manuel Mendez <manny@packet.com>

mkinitfs has gained similar functionality as originally implemented with packet_bootdev_mac so lets just use that and minimize the patch diff. mkinitfs also learned to not blindly use first eth device. The implementation is different that what we currently do, but seems to be fine. It doesn't really matter though since we always set packet_bootdev_mac when booting with boots which alpine always honors. Signed-off-by: Manuel Mendez <manny@packet.com>

Some servers / nics are finicky and may not be ready at this point in boot yet. Signed-off-by: Manuel Mendez <manny@packet.com>

We need this because we will be embedding packages from main and community (docker). Signed-off-by: Manuel Mendez <manny@packet.com>

Signed-off-by: Manuel Mendez <manny@packet.com>

apk will only use files relative to `--root` so we need the embedded cached packages and index files within $sysroot. Also need to override ALPINE_REPO otherwise apk won't use the cached packages. Signed-off-by: Manuel Mendez <manny@packet.com>

Similar to the setup needed for cached packages but for our custom built apks. Signed-off-by: Manuel Mendez <manny@packet.com>

Signed-off-by: Manuel Mendez <manny@packet.com>

This is the init used in alpine-3.12 + our changes. Signed-off-by: Manuel Mendez <manny@packet.com>

Mmmm fresh out the oven and smells delicious. Signed-off-by: Manuel Mendez <manny@packet.com>

Matches our other Dockerfiles and helps differentiate between commands in `&&` chain and args for them. Signed-off-by: Manuel Mendez <manny@packet.com>

The json output is broken on latest tagged release used in 3.12. Signed-off-by: Manuel Mendez <manny@packet.com>

…pine py3-pip brings in dependencies that conflict with our own, and don't seem to actually be needed. If those dependencies are in the system, py3-pip will be more than happy to use them. Later when we delete the build-deps virtual package those packages go away too and now we can't start up. By using a non-package pip we not only avoid competing package managers , but also don't come across issues with dependencies as pip bundles its dependencies. Signed-off-by: Manuel Mendez <manny@packet.com>

rgl · 2020-08-18T20:36:42Z

ci/Dockerfile

+        && \
+    mv /etc/apk/repositories /etc/apk/repositories.old && \
+    apk add --no-cache --update --upgrade \
+        --repository=http://dl-cdn.alpinelinux.org/alpine/v3.11/main \


I think this needs to be changed to v3.12 (maybe do a global search and replace just to check whether this does not appear elsewhere).

no this is to get lshw from 3.11 explicitly. v3.12's lshw doesn't output correct json. I have not been able to track down the commit that broke lshw.

Oh I see :-(

Its only used in tests so not a big deal imo. Looks like upstream has fixed the json output though so I'll look into PR'ing patches to aports.

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/11607 in case you're interested @rgl

mmlb requested review from splaspood, nathangoulding and mikemrm July 17, 2020 21:38

mmlb marked this pull request as draft July 17, 2020 21:42

mmlb requested a review from ScottGarman July 17, 2020 22:03

mikemrm reviewed Jul 17, 2020

View reviewed changes

mrmrcoleman added this to In Progress in Issues List Jul 21, 2020

This was referenced Jul 27, 2020

Alpine 3.12 #73

Closed

Configure Alpine networking via ip kernel command line tinkerbell/smee#62

Closed

Integrate boot file make into top level #79

Merged

mmlb added 7 commits August 18, 2020 10:24

Upgrade x86_64 Alpine containers to 3.12

09c3aa2

Needed to upgrade deps since `six` is a disutils package and pip was having trouble with it. I manually held `grpcio` back since it fails to build in Alpine 3.12 [1]. [1]: grpc/grpc#23742 Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Fix whitespace

ca900b4

Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Update kernel to latest LTS.

2510506

Going to need a newer kernel for the Pensando NICs and might as well use the latest LTS release. Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Install linux-firmware from aports checkout

92cb569

Its not great for reproducibility to install from edge at any random point in time of the repo. Since we already have a checkout of aports, might as well use that. Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Enable IONIC (pensando) driver

ef32119

Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Install eclypsium package just like the others

9311d2a

Signed-off-by: Manuel Mendez <manny@packet.com>

mmlb marked this pull request as ready for review August 18, 2020 16:12

mmlb requested review from a team and mikemrm August 18, 2020 16:12

mmlb added the ready-to-merge Signal to Mergify to merge the PR. label Aug 18, 2020

mmlb removed request for ScottGarman and nathangoulding August 18, 2020 17:38

mikemrm previously approved these changes Aug 18, 2020

View reviewed changes

mmlb and others added 5 commits August 18, 2020 14:40

installer/alpine: Build each apk in separate RUN

1ab00e0

For better caching/re-runs. There is no point in agonizing over docker layers as we do not ship this image anywhere. Signed-off-by: Manuel Mendez <manny@packet.com>

Add init-vanilla-3.12 from mkintfs 3.4.5 source

58aac79

mkintfs is installed from v3.4.5 in the Dockefile, which maps to https://gitlab.alpinelinux.org/alpine/mkinitfs/-/raw/v3.4.5/initramfs-init.in Signed-off-by: Manuel Mendez <manny@packet.com>

mmlb added 21 commits August 18, 2020 14:40

Add init-x86_64.next

8e2735f

Fetched from https://raw.githubusercontent.com/alpinelinux/mkinitfs/v3.4.5/initramfs-init.in and replaced @Version@ with tag as is done in the Makefile. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Fix curly brace use withing variable pattern substitution

dca3503

Makes a bunch of tools happier, shfmt, vim syntax highlighting. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Add --no-progress to all apk add commands

1b147c1

Avoids spraying console with uninteresting progress output. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Explicitly mount cgroup fs

1c956c5

Otherwise docker won't start up. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Retry nic reset and dhcp config a few times

f4e34c1

Some servers / nics are finicky and may not be ready at this point in boot yet. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Handle mulitiple repos in ALPINE_REPO

c24129f

We need this because we will be embedding packages from main and community (docker). Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Add required packet args

ef13a24

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Add packet_state handling

af0285c

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Add packet_base_url to myopts

75da558

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Fetch and install $packet_action scripts

ae0fcb1

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Setup KOPT_modules for eclypsium at deprovisioning time

76b8ee8

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Unconditionally run apk with --no-network

3fa701d

Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Configure apk to use our local repo

a018880

Similar to the setup needed for cached packages but for our custom built apks. Signed-off-by: Manuel Mendez <manny@packet.com>

init.next: Set KOPT_pkgs with packages we need installed

14a321e

Signed-off-by: Manuel Mendez <manny@packet.com>

installer/alpine: Replace init-x86_64 with init-x86_64.next

7e4e7d5

This is the init used in alpine-3.12 + our changes. Signed-off-by: Manuel Mendez <manny@packet.com>

Add freshly built Alpine 3.12 boot files

ef35f88

Mmmm fresh out the oven and smells delicious. Signed-off-by: Manuel Mendez <manny@packet.com>

Fix apk packages alignment

e282197

Matches our other Dockerfiles and helps differentiate between commands in `&&` chain and args for them. Signed-off-by: Manuel Mendez <manny@packet.com>

ci: Use lshw from 3.11

4c82d76

The json output is broken on latest tagged release used in 3.12. Signed-off-by: Manuel Mendez <manny@packet.com>

mmlb dismissed mikemrm’s stale review via 0ec277c August 18, 2020 18:41

mmlb requested a review from mikemrm August 18, 2020 18:42

mikemrm approved these changes Aug 18, 2020

View reviewed changes

mmlb removed the request for review from splaspood August 18, 2020 18:44

Merge branch 'master' into alpine-3.12

c713fe9

mergify bot merged commit e902e83 into tinkerbell:master Aug 18, 2020

Issues List automation moved this from In Progress to Just shipped Aug 18, 2020

mmlb deleted the alpine-3.12 branch August 18, 2020 19:43

rgl reviewed Aug 18, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alpine 3.12 #72

Alpine 3.12 #72

mmlb commented Jul 17, 2020 •

edited

Loading

mikemrm Jul 17, 2020

mmlb Jul 19, 2020

mmlb Aug 6, 2020

mikemrm Jul 17, 2020

mmlb Jul 19, 2020

mikemrm Jul 17, 2020

mmlb Jul 19, 2020

mikemrm Jul 17, 2020

mmlb Jul 19, 2020

mikemrm left a comment

rgl Aug 18, 2020

mmlb Aug 18, 2020

rgl Aug 19, 2020

mmlb Aug 19, 2020

mmlb Aug 19, 2020

Alpine 3.12 #72

Alpine 3.12 #72

Conversation

mmlb commented Jul 17, 2020 • edited Loading

Description

Why is this needed

How Has This Been Tested?

How are existing users impacted? What migration steps/scripts do we need?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikemrm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmlb commented Jul 17, 2020 •

edited

Loading