Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix qemu binfmt image is pulled during instance startup #1231

Merged
merged 12 commits into from
Oct 4, 2023
Merged

Fix qemu binfmt image is pulled during instance startup #1231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
644d00c
Never pull docker image on startup
triarius Oct 1, 2023
9677cf5
List image on binfmt fail
triarius Oct 1, 2023
50b2358
Write more env vars to lib file and give a better description of why …
triarius Oct 2, 2023
747faec
Fix docker run does not recognize previously pulled image
triarius Oct 2, 2023
f91a2d9
Fix sourcing
triarius Oct 2, 2023
36ab799
Binfmt before userns
triarius Oct 2, 2023
473a071
Log startup of each script
triarius Oct 2, 2023
5b35baf
Remove install from systemd services used by timers
triarius Oct 2, 2023
2192ee6
Delay GC until after docker setup
triarius Oct 2, 2023
6bf4d79
Cleanup error message when docker run fails
triarius Oct 3, 2023
13c81f5
Remove sudo use in startup script
triarius Oct 3, 2023
b8a728f
Make multi-line conditional more clear
triarius Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 24 additions & 10 deletions packer/linux/conf/bin/bk-configure-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,31 @@ trap '[[ $? = 0 ]] && on_exit' EXIT
# See https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee -a /var/log/elastic-stack.log | logger -t user-data -s 2>/dev/console) 2>&1

echo "Starting ${BASH_SOURCE[0]}..."

echo Reading variables from AMI creation...
echo Sourcing /usr/local/lib/bk-configure-docker.sh...
echo This file is written by the scripts in packer/scripts.
echo Note that the path is /usr/local/lib, not /usr/local/bin.
echo Contents of /usr/local/lib/bk-configure-docker.sh:
cat /usr/local/lib/bk-configure-docker.sh
# shellcheck disable=SC1091
source /usr/local/lib/bk-configure-docker.sh

echo Installing qemu binfmt for multiarch...
if ! docker run \
--privileged \
--userns=host \
--pull=never \
--rm \
"tonistiigi/binfmt@${QEMU_BINFMT_DIGEST}" \
--install all
then
echo Failed to install binfmt.
echo Avaliable docker images:
docker image ls
exit 1
fi

if [[ "${DOCKER_USERNS_REMAP:-false}" == "true" ]]; then
echo Configuring user namespace remapping...

Expand Down Expand Up @@ -72,17 +92,11 @@ cat <<<"$(jq \
/etc/docker/daemon.json \
)" >/etc/docker/daemon.json

# See https://docs.docker.com/build/building/multi-platform/
echo Installing qemu binfmt for multiarch...
docker run \
--privileged \
--userns=host \
--rm \
"tonistiigi/binfmt:${QEMU_BINFMT_TAG}" \
--install all

echo Cleaning up docker images...
systemctl start docker-low-disk-gc.service

echo Enabling docker-gc timers...
systemctl enable docker-gc.timer docker-low-disk-gc.timer

echo Restarting docker daemon...
systemctl restart docker
2 changes: 2 additions & 0 deletions packer/linux/conf/bin/bk-install-elastic-stack.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ trap '[[ $? = 0 ]] && on_exit' EXIT
# See https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee -a /var/log/elastic-stack.log | logger -t user-data -s 2>/dev/console) 2>&1

echo "Starting ${BASH_SOURCE[0]}..."

# This needs to happen first so that the error reporting works
token=$(curl -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 60" --fail --silent --show-error --location http://169.254.169.254/latest/api/token)
INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $token" --fail --silent --show-error --location http://169.254.169.254/latest/meta-data/instance-id)
Expand Down
2 changes: 2 additions & 0 deletions packer/linux/conf/bin/bk-mount-instance-storage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ trap '[[ $? = 0 ]] && on_exit' EXIT
# See https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee -a /var/log/elastic-stack.log | logger -t user-data -s 2>/dev/console) 2>&1

echo "Starting ${BASH_SOURCE[0]}..."

# Mount instance storage if we can
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

Expand Down
3 changes: 0 additions & 3 deletions packer/linux/conf/docker/systemd/docker-gc.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,3 @@ Wants=docker-gc.timer
[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-gc

[Install]
WantedBy=multi-user.target
3 changes: 0 additions & 3 deletions packer/linux/conf/docker/systemd/docker-low-disk-gc.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,3 @@ Wants=docker-low-disk-gc.timer
[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-low-disk-gc

[Install]
WantedBy=multi-user.target
15 changes: 12 additions & 3 deletions packer/linux/scripts/install-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ echo "Adding docker systemd timers..."
sudo cp /tmp/conf/docker/scripts/* /usr/local/bin
sudo cp /tmp/conf/docker/systemd/docker-* /etc/systemd/system
sudo chmod +x /usr/local/bin/docker-*
sudo systemctl daemon-reload
sudo systemctl enable docker-gc.timer docker-low-disk-gc.timer

echo "Installing docker buildx..."
DOCKER_CLI_DIR=/usr/libexec/docker/cli-plugins
Expand All @@ -47,11 +45,22 @@ sudo cp /tmp/conf/bin/docker-compose /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version

# Writing QEMU container version info to /usr/local/lib/bk-configure-docker.sh.
# We only pull this image when we build the AMI. It will be run in
# /usr/local/bin/bk-configure-docker.sh, but it needs to know the image digest
# to make sure it does not pull in another image instead.
# NOTE: the executable file is in /usr/local/bin and it sources as file of the
# same name in /usr/local/lib. These are not the same file.
# See https://docs.docker.com/build/building/multi-platform/

echo Contents of /usr/local/lib/bk-configure-docker.sh:
cat <<'EOF' | sudo tee -a /usr/local/lib/bk-configure-docker.sh
QEMU_BINFMT_VERSION=7.0.0-28
QEMU_BINFMT_DIGEST=sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55
QEMU_BINFMT_TAG="qemu-v${QEMU_BINFMT_VERSION}@${QEMU_BINFMT_DIGEST}"
EOF
# shellcheck disable=SC1091
source /usr/local/lib/bk-configure-docker.sh
sudo mkdir -p /usr/local/lib
echo "QEMU_BINFMT_TAG=\"$QEMU_BINFMT_TAG\"" | sudo tee -a /usr/local/lib/bk-configure-docker.sh
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When images are pulled with a tag and digest, docker does not seem to keep the tag. Try it with

$ docker pull alpine:3.18.4@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978
$ docker image ls
REPOSITORY                   TAG             IMAGE ID       CREATED         SIZE
alpine                       <none>          8ca4688f4f35   4 days ago      7.33MB

I don't think it affects docker run though 🤷‍♂️.

echo Pulling qemu binfmt for multiarch...
sudo docker pull "tonistiigi/binfmt:${QEMU_BINFMT_TAG}"