Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix cross compilation for arm platforms #43474

Merged
merged 4 commits into from
Apr 14, 2022
+84 −7
Merged

fix cross compilation for arm platforms #43474

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c2dfb60
ci: update upload-artifact and download-artifact actions to v3
crazy-max Apr 9, 2022
d730d55
ci: set shorter retention period for artifacts
crazy-max Apr 9, 2022
12558c8
fix cross compilation for arm platforms
crazy-max Apr 10, 2022
1700492
ci: add cross job in ci workflow
crazy-max Apr 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 56 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,62 @@ jobs:
*.cache-from=type=gha,scope=build-${{ matrix.target }}
*.cache-to=type=gha,scope=build-${{ matrix.target }}
-
name: Upload build artifacts
uses: actions/upload-artifact@v2
name: Upload artifacts
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.target }}
path: ${{ env.BUNDLES_OUTPUT }}/${{ matrix.target }}-daemon/*
path: ${{ env.BUNDLES_OUTPUT }}
if-no-files-found: error
retention-days: 7

cross:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove the cross stage from Jenkins once this is merged?

remove --build-arg CROSS=true from

moby/Jenkinsfile

Line 90 in c687298

sh 'docker build --force-rm --build-arg APT_MIRROR --build-arg CROSS=true -t docker:${GIT_COMMIT} .'

remove the cross stage:

moby/Jenkinsfile

Lines 167 to 179 in c687298

stage("Cross") {
steps {
sh '''
docker run --rm -t --privileged \
-v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
--name docker-pr$BUILD_NUMBER \
-e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-e DOCKER_GRAPHDRIVER \
docker:${GIT_COMMIT} \
hack/make.sh cross
'''
}
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove the CROSS and DOCKER_CROSSPLATFORMS args in a follow-up and rely only on --platform like docker/cli, buildx so packaging will be easier and also we could remove the cross logic in the .binary script. But that needs some work to align the dependencies in the dockerfile.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make sure it can be run outside of the Dockerfile / container as well; it's possible there's people using the scripts to build/cross compile directly on the host.

(I guess xx can be used in that situation as well, but may need either extra instructions or a script to set it up)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need a non-Dockerfile "cross-all" type target if we have no use for it ourselves. The script for the single arch would run for any target arch inside Dockerfile(and cross-all is just --platform list). To run the same script locally it is user's responsibility that they have a cross toolchain for the target arch they are setting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's possible there's people using the scripts to build/cross compile directly on the host.

I don't know of distros that would use these scripts for cross-compilation, but I do know at least some distros definitely build their packages outside of containers/in their own environments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's continue that discussion when that change is actually made (this PR does not yet make that change)

runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm/v5
- linux/arm/v6
- linux/arm/v7
- linux/arm64
- linux/ppc64le
- linux/s390x
- windows/amd64
crazy-max marked this conversation as resolved.
Show resolved Hide resolved
- windows/arm64
steps:
-
name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
-
name: Prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
mkdir -p autogen
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Build
uses: docker/bake-action@v1
with:
targets: cross
set: |
*.cache-from=type=gha,scope=cross-${{ env.PLATFORM_PAIR }}
*.cache-to=type=gha,scope=cross-${{ env.PLATFORM_PAIR }}
env:
DOCKER_CROSSPLATFORMS: ${{ matrix.platform }}
-
name: Upload artifacts
uses: actions/upload-artifact@v2
with:
name: cross-${{ env.PLATFORM_PAIR }}
path: ${{ env.BUNDLES_OUTPUT }}
if-no-files-found: error
retention-days: 7

test-buildkit:
needs:
Expand Down Expand Up @@ -91,8 +141,8 @@ jobs:
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Download build artifacts
uses: actions/download-artifact@v2
name: Download binary artifacts
uses: actions/download-artifact@v3
with:
name: binary
path: ./buildkit/build/moby/
Expand All @@ -110,7 +160,7 @@ jobs:
env:
CONTEXT: "."
TEST_DOCKERD: "1"
TEST_DOCKERD_BINARY: "./build/moby/dockerd"
TEST_DOCKERD_BINARY: "./build/moby/binary-daemon/dockerd"
TESTPKGS: "${{ matrix.pkg }}"
TESTFLAGS: "-v --parallel=1 --timeout=30m --run=//worker=dockerd$"
SKIP_INTEGRATION_TESTS: "${{ matrix.skip-integration-tests }}"
Expand Down
13 changes: 13 additions & 0 deletions docker-bake.hcl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
variable "BUNDLES_OUTPUT" {
default = "./bundles"
}
variable "DOCKER_CROSSPLATFORMS" {
default = ""
}

target "_common" {
args = {
BUILDKIT_CONTEXT_KEEP_GIT_DIR = 1
APT_MIRROR = "cdn-fastly.deb.debian.org"
}
}

Expand All @@ -22,3 +26,12 @@ target "dynbinary" {
inherits = ["binary"]
target = "dynbinary"
}

target "cross" {
inherits = ["binary"]
args = {
CROSS = "true"
DOCKER_CROSSPLATFORMS = DOCKER_CROSSPLATFORMS
}
target = "cross"
}
16 changes: 15 additions & 1 deletion hack/make/.binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,25 @@ hash_files() {
;;
linux/arm)
case "${GOARM}" in
5 | "")
5)
export CC="${CC:-arm-linux-gnueabi-gcc}"
export CGO_ENABLED=1
export CGO_CFLAGS="-march=armv5t"
export CGO_CXXFLAGS="-march=armv5t"
;;
6)
export CC="${CC:-arm-linux-gnueabi-gcc}"
export CGO_ENABLED=1
export CGO_CFLAGS="-march=armv6"
export CGO_CXXFLAGS="-march=armv6"
;;
7)
export CC="${CC:-arm-linux-gnueabihf-gcc}"
export CGO_ENABLED=1
export CGO_CFLAGS="-march=armv7-a"
export CGO_CXXFLAGS="-march=armv7-a"
;;
*)
export CC="${CC:-arm-linux-gnueabihf-gcc}"
export CGO_ENABLED=1
;;
Expand Down