Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hyperkube-base: perform magic COPY incantation to make apt-get update work properly #67215

Closed
wants to merge 1 commit into from

Conversation

ixdy
Copy link
Member

@ixdy ixdy commented Aug 9, 2018

This PR seriously bothers me.

I've attempted to build a new release of the debian-hyperkube-base images (after #67026), but I'm unable to cross-build any of the non-amd64 images, while the debian-iptables images build without issue.

The errors were around signatures on the apt repositories:

Step 2/6 : RUN echo 2e285e3b-2127-4c83-af22-db258c7ae7ce>/dev/null && clean-install     bash
 ---> Running in 8169361d8294
Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:1 http://security.debian.org/debian-security stretch/updates InRelease
Get:3 http://security.debian.org/debian-security stretch/updates/main armhf Packages [372 kB]
Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:4 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Ign:4 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease
Get:6 http://cdn-fastly.deb.debian.org/debian stretch-updates/main armhf Packages [5112 B]
Get:5 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B]
Ign:7 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg
Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main armhf Packages [6927 kB]
Fetched 7610 kB in 5s (1465 kB/s)
Reading package lists...
W: GPG error: http://security.debian.org/debian-security stretch/updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_stretch_updates_InRelease
W: The repository 'http://security.debian.org/debian-security stretch/updates InRelease' is not signed.
W: GPG error: http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_stretch-updates_InRelease
W: The repository 'http://deb.debian.org/debian stretch-updates InRelease' is not signed.
W: GPG error: http://cdn-fastly.deb.debian.org/debian stretch Release: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_stretch_Release
W: The repository 'http://deb.debian.org/debian stretch Release' is not signed.
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  bash-doc
Recommended packages:
  bash-completion
The following NEW packages will be installed:
  bash
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 1371 kB of archives.
After this operation, 5573 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  bash
E: There were unauthenticated packages and -y was used without --allow-unauthenticated
The command '/bin/sh -c echo 2e285e3b-2127-4c83-af22-db258c7ae7ce>/dev/null && clean-install     bash' returned a non-zero code: 100

Through trial and error, I discovered that the only difference between the debian-iptables image builds and debian-hyperkube-base builds were that the iptables builds download the qemu-ARCH-static binaries and COPY them into the container.

The qemu-ARCH-static binaries already exist in the base image (debian-base), and the version, permissions, sha256sum, and other metadata are unchanged. Somehow copying it again (via the COPY directive) matters.

If I COPY to some other path, it doesn't work.

For even more spookiness, building the hyperkube-base image from debian-base:0.3 works without issue.

If I remove the COPY lines from the debian-iptables Dockerfile, I can't build that image anymore.

Release note:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 9, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ixdy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 9, 2018
@ixdy
Copy link
Member Author

ixdy commented Aug 9, 2018

/assign @tallclair @satyasm
cc @mkumatag

@k8s-ci-robot
Copy link
Contributor

@ixdy: GitHub didn't allow me to assign the following users: satyasm.

Note that only kubernetes members and repo collaborators can be assigned.
For more information please see the contributor guide

In response to this:

/assign @tallclair @satyasm
cc @mkumatag

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ixdy
Copy link
Member Author

ixdy commented Aug 9, 2018

cc @dims

@ixdy
Copy link
Member Author

ixdy commented Aug 9, 2018

/close

@BenTheElder figured out what was wrong (after I started at this for an hour+):

$ docker run --rm -ti --entrypoint ls k8s.gcr.io/debian-base-arm:0.3 -l /usr/bin/qemu-arm-static
-rwxr-xr-x 1 root root 3506464 Jun  7  2017 /usr/bin/qemu-arm-static
$ docker run --rm -ti --entrypoint ls k8s.gcr.io/debian-base-arm:0.3.1 -l /usr/bin/qemu-arm-static
-rwxr-x--- 1 root root 3506464 Jun  7  2017 /usr/bin/qemu-arm-static

I'm guessing when 0.3.1 was built, a restrictive umask was in place, so the binary was installed without the proper permissions. I'll fix the Dockerfile in for debian-base and push a 0.3.2.

@tallclair
Copy link
Member

awesome, thanks for investigating. Once that's fixed, can we remove the duplicate copy from debian-iptables too?

@ixdy
Copy link
Member Author

ixdy commented Aug 9, 2018

yep, I folded that into my fix PR (#67222).

k8s-github-robot pushed a commit that referenced this pull request Aug 10, 2018
Automatic merge from submit-queue (batch tested with PRs 67058, 67083, 67220, 67222, 67209). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix permissions of qemu-ARCH-static in debian-base and other images

**What this PR does / why we need it**: proper fix for the issue I found in #67215. Some machines (like apparently workstations at Google) have a restrictive umask, so the `qemu-ARCH-static` binaries were getting installed in images without world read/execute permissions, causing utilities like `apt-get` to fail.

There was also a duplicate download/install of these binaries for `debian-iptables`, which further confused the issue. I've since removed that duplicate installation.

Many thanks to @BenTheElder for asking the right question to get me to look at the permissions again.

I haven't pushed any images yet. After merge, I'll build/promote `debian-base:0.3.2`, then update everything to use it, then push some more images, write some more PRs, ...

**Release note**:

```release-note
NONE
```
/assign @tallclair
@ixdy ixdy deleted the debian-hyperkube-base-qemu branch August 12, 2018 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants