Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Differentiate build process for non-amd platforms #82778

Merged
merged 2 commits into from Feb 20, 2021

Conversation

chendave
Copy link
Member

@chendave chendave commented Sep 17, 2019

  • reset binfmt_misc is needn't when the building platform is non-amd64 and the
    target arch is the same as building platform

  • non-amd64 platform doesn't supported cross-build well, and there is no binary of
    qemu-user-static able to do that, and thus skip the cross-build on non-amd64
    platform.

Here is what it looks like for apparmor-loader build on ARM64.

./image-util.sh build apparmor-loader
Building image for apparmor-loader OS/ARCH: linux/amd64...
make[1]: Entering directory '/go/src/k8s.io/kubernetes/test/images/apparmor-loader'
../image-util.sh bin loader
go: downloading k8s.io/klog v1.0.0
go: extracting k8s.io/klog v1.0.0
go: finding k8s.io/klog v1.0.0
make[1]: Leaving directory '/go/src/k8s.io/kubernetes/test/images/apparmor-loader'
/go/src/k8s.io/kubernetes/_tmp/test-images-build.g6iRGF /go/src/k8s.io/kubernetes/test/images
skip cross-build amd64 on non-supported platform aarch64.
...
Building image for apparmor-loader OS/ARCH: linux/arm64...
make[1]: Entering directory '/go/src/k8s.io/kubernetes/test/images/apparmor-loader'
../image-util.sh bin loader
...
Successfully built efc23c2ab906
Successfully tagged gcr.io/kubernetes-e2e-test-images/apparmor-loader:1.1-linux-arm64

Signed-off-by: Dave Chen dave.chen@arm.com

What type of PR is this?
/kind bug

Which issue(s) this PR fixes:
Fixes #82775
Fixes #81920
Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 17, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @chendave. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 17, 2019
@chendave
Copy link
Member Author

cc @BenTheElder @mkumatag @dims

@BenTheElder
Copy link
Member

sorry I'm behind on reviews
/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 19, 2019
@BenTheElder
Copy link
Member

broadly the idea SGTM
cc @cblecker @dims

@chendave
Copy link
Member Author

@BenTheElder Thanks for picking this up.

@mkumatag
Copy link
Member

broadly the idea SGTM
cc @cblecker @dims

+1, will review once I get some time over the weekend, need good amount of testing as well.

@chendave
Copy link
Member Author

@mkumatag thanks for your time on this.

@chendave
Copy link
Member Author

poke @mkumatag , could you take a look at this pls?

@mkumatag
Copy link
Member

poke @mkumatag , could you take a look at this pls?

well,! I'm busy today, I will take a look in the next couple of days!

test/images/image-util.sh Outdated Show resolved Hide resolved
test/images/image-util.sh Outdated Show resolved Hide resolved
test/images/image-util.sh Outdated Show resolved Hide resolved
@chendave chendave force-pushed the split_build_scripts branch 2 times, most recently from 010788f to ba9fc0a Compare November 26, 2019 05:31
@chendave
Copy link
Member Author

@mkumatag all comments addressed, could you pls take another look? thank you!

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 24, 2020
@chendave
Copy link
Member Author

chendave commented Jan 29, 2021

@chendave please rebase and we'll look again?

Done

@chendave
Copy link
Member Author

/test pull-kubernetes-integration

@chendave
Copy link
Member Author

@joakimr-axis thanks for the review, all fixed, PTAL again?

@chendave
Copy link
Member Author

/retest

@joakimr-axis
Copy link
Contributor

@joakimr-axis thanks for the review, all fixed, PTAL again?

Looks really nice now, IMO!

@dims
Copy link
Member

dims commented Feb 19, 2021

verified that this works! here's the log:

``` ubuntu@ip-172-31-81-16:~/go/src/k8s.io/kubernetes/test/images$ make WHAT=apparmor-loader ./image-util.sh build apparmor-loader "docker" Building image for apparmor-loader OS/ARCH: linux/amd64... make[1]: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ../image-util.sh bin loader Unable to find image 'golang:1.15.7' locally 1.15.7: Pulling from library/golang 6e5587ff5efa: Already exists 439dbbb05ea0: Already exists 3b89c8b4e5b2: Already exists 4a53f70a43c3: Already exists d56052def468: Already exists 0371b45dab95: Already exists d81d5ca3d230: Already exists Digest: sha256:c161abf0cde3969e05f6914a86cab804b2b0df515f4ff9570475b25547ba7959 Status: Downloaded newer image for golang:1.15.7 make[1]: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ~/go/src/k8s.io/kubernetes/_tmp/test-images-build.X4kmL8 ~/go/src/k8s.io/kubernetes/test/images skip cross-build amd64 on non-supported platform aarch64. ~/go/src/k8s.io/kubernetes/test/images Building image for apparmor-loader OS/ARCH: linux/arm... make[1]: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ../image-util.sh bin loader make[1]: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ~/go/src/k8s.io/kubernetes/_tmp/test-images-build.UmKtdN ~/go/src/k8s.io/kubernetes/test/images skip cross-build arm on non-supported platform aarch64. ~/go/src/k8s.io/kubernetes/test/images Building image for apparmor-loader OS/ARCH: linux/arm64... make[1]: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ../image-util.sh bin loader make[1]: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader' ~/go/src/k8s.io/kubernetes/_tmp/test-images-build.zU1Lsj ~/go/src/k8s.io/kubernetes/test/images #1 [internal] load build definition from Dockerfile #1 sha256:7c4699ae537131bb0f2c400d16759e5047cca38947fd03a15438b5132a32597d #1 transferring dockerfile: 1.02kB done #1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:8c68f4a7464b8841954974b816960c33ed1f036c5d10042bd8552f42c95de2ac
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/arm64v8/alpine:3.8
#3 sha256:b7d89063be271a1a2b059b7075ff01a4c587be4d21506d6af2b491566145b0ed
#3 DONE 0.4s

#6 [internal] load build context
#6 sha256:d9a7cf7b351564b69bf9c40cc1de9afc7d6c80c6f6b95b686d3c75b9050efbcd
#6 transferring context: 2.07MB 0.1s done
#6 DONE 0.1s

#4 [1/3] FROM docker.io/arm64v8/alpine:3.8@sha256:e802987f152d7826cf929ad4999fb3bb956ce7a30966aeb46c749f9120eaf22c
#4 sha256:24f3a7ff9a87ea1fb3fb3f83e192387ed885bc1d792452c6118fb8f701ece51c
#4 resolve docker.io/arm64v8/alpine:3.8@sha256:e802987f152d7826cf929ad4999fb3bb956ce7a30966aeb46c749f9120eaf22c done
#4 sha256:e802987f152d7826cf929ad4999fb3bb956ce7a30966aeb46c749f9120eaf22c 528B / 528B done
#4 sha256:b22edbe95d11980cbd47579189e8e6382dbb39629ca636f2a3b2741fae23bf1d 1.51kB / 1.51kB done
#4 sha256:788aef77d06ba65af872cf0c2da5b49362f6c05a5c8d1f8ceb4fd8b070e56876 2.10MB / 2.10MB 0.1s done
#4 extracting sha256:788aef77d06ba65af872cf0c2da5b49362f6c05a5c8d1f8ceb4fd8b070e56876 0.1s done
#4 DONE 0.2s

#5 [2/3] RUN apk add apparmor libapparmor --update-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ --allow-untrusted && apk add --no-cache musl>1.1.20 --repository http://dl-cdn.alpinelinux.org/alpine/edge/main/
#5 sha256:081c3b71e868ada219469ec8783caff6932f1a8d69cd77bbc74c1e78dc405fd8
#5 0.221 fetch http://dl-cdn.alpinelinux.org/alpine/edge/testing/aarch64/APKINDEX.tar.gz
#5 0.354 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/aarch64/APKINDEX.tar.gz
#5 0.491 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/aarch64/APKINDEX.tar.gz
#5 0.657 (1/9) Installing ncurses-terminfo-base (6.1_p20180818-r1)
#5 0.667 (2/9) Installing ncurses-terminfo (6.1_p20180818-r1)
#5 0.823 (3/9) Installing ncurses-libs (6.1_p20180818-r1)
#5 0.830 (4/9) Installing readline (7.0.003-r0)
#5 0.836 (5/9) Installing bash (4.4.19-r1)
#5 0.855 Executing bash-4.4.19-r1.post-install
#5 0.857 (6/9) Installing libintl (0.19.8.1-r2)
#5 0.867 (7/9) Installing apparmor (2.13.4-r0)
#5 1.269 (8/9) Installing sed (4.4-r2)
#5 1.280 (9/9) Installing libapparmor (2.13.4-r0)
#5 1.451 Executing busybox-1.28.4-r3.trigger
#5 1.456 OK: 14 MiB in 22 packages
#5 1.490 fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
#5 1.621 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/aarch64/APKINDEX.tar.gz
#5 1.674 fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/aarch64/APKINDEX.tar.gz
#5 1.759 (1/2) Upgrading musl (1.1.19-r11 -> 1.2.2-r1)
#5 1.778 (2/2) Upgrading musl-utils (1.1.19-r11 -> 1.2.2-r1)
#5 1.790 Executing busybox-1.28.4-r3.trigger
#5 1.793 OK: 14 MiB in 22 packages
#5 DONE 2.0s

#7 [3/3] ADD loader /usr/bin/loader
#7 sha256:4cec41f83d234b0aa4fd59ba93052992378d07358ce36254f65b4e6ad0a253b1
#7 DONE 0.1s

#8 exporting to image
#8 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#8 exporting layers
#8 exporting layers 0.3s done
#8 writing image sha256:c2954f52051d8a05353a59c559e3199442ea903a2dc54ccc1dde9fe13010e2f7 done
#8 naming to gcr.io/kubernetes-e2e-test-images/apparmor-loader:1.3-linux-arm64 done
#8 DONE 0.3s
~/go/src/k8s.io/kubernetes/test/images
Building image for apparmor-loader OS/ARCH: linux/ppc64le...
make[1]: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader'
../image-util.sh bin loader
make[1]: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes/test/images/apparmor-loader'
~/go/src/k8s.io/kubernetes/_tmp/test-images-build.gBZZ9I ~/go/src/k8s.io/kubernetes/test/images
skip cross-build ppc64le on non-supported platform aarch64.
~/go/src/k8s.io/kubernetes/test/images

ubuntu@ip-172-31-81-16:~/go/src/k8s.io/kubernetes/test/images$ uname -a
Linux ip-172-31-81-16 5.4.0-1037-aws #39-Ubuntu SMP Thu Jan 14 02:53:31 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

ubuntu@ip-172-31-81-16:~/go/src/k8s.io/kubernetes/test/images$ cat /etc/release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

</details>

@dims
Copy link
Member

dims commented Feb 19, 2021

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 19, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chendave, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 19, 2021
@dims
Copy link
Member

dims commented Feb 19, 2021

/retest

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@pacoxu
Copy link
Member

pacoxu commented Feb 20, 2021

/test pull-kubernetes-e2e-kind
open #99252 for the volume flake

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 0a3cd56 into kubernetes:master Feb 20, 2021
@chendave
Copy link
Member Author

@dims thanks for verifying this PR, it's great to see this got merged eventually and release me from this long standing changing request.

@chendave chendave deleted the split_build_scripts branch February 22, 2021 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet