Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to cross-compiled docker containers & container build #1571

Closed
wants to merge 5 commits into from
Closed

Switch to cross-compiled docker containers & container build #1571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8f3830a
Use docker buildx to cross-build multi-platform container images
holdenk Jun 29, 2021
f88b586
Fix the add path for gen-admission-secret.sh
holdenk Jun 29, 2021
f29b2ed
Make the default behaviour single image so we don't break the hacks s…
holdenk Jun 29, 2021
0568eb3
Clarify the build instructions (e.g. registery/prefix)
holdenk Jul 8, 2021
2a427e0
Switch it back to go 1.14
holdenk Jul 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
29 changes: 10 additions & 19 deletions Makefile
View file
Open in desktop
Expand Up @@ -20,6 +20,7 @@ CRD_OPTIONS ?= "crd:crdVersions=v1,generateEmbeddedObjectMeta=true"
CC ?= "gcc"
SUPPORT_PLUGINS ?= "no"
CRD_VERSION ?= v1
BUILDX_OUTPUT_TYPE ?= "docker"

# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
ifeq (,$(shell go env GOBIN))
Expand Down Expand Up @@ -52,6 +53,10 @@ else
REL_OSARCH=linux/$(OSARCH)
endif

# Run make images DOCKER_PLATFORMS="linux/amd64,linux/arm64" BUILDX_OUTPUT_TYPE=registry IMAGE_PREFIX=[yourregistry]/vc to push multi-platform
DOCKER_PLATFORMS ?= "${REL_OSARCH}"


include Makefile.def

.EXPORT_ALL_VARIABLES:
Expand Down Expand Up @@ -84,30 +89,16 @@ image_bins: init
if [ ${SUPPORT_PLUGINS} = "yes" ];then\
CC=${CC} CGO_ENABLED=1 $(GOBIN)/gox -osarch=${REL_OSARCH} -ldflags ${LD_FLAGS} -output ${BIN_DIR}/${REL_OSARCH}/vc-scheduler ./cmd/scheduler;\
else\
CC=${CC} CGO_ENABLED=0 $(GOBIN)/gox -osarch=${REL_OSARCH} -ldflags ${LD_FLAGS} -output ${BIN_DIR}/${REL_OSARCH}/vc-scheduler ./cmd/scheduler;\
fi;
CC=${CC} CGO_ENABLED=0 $(GOBIN)/gox -osarch=${REL_OSARCH} -ldflags ${LD_FLAGS} -output ${BIN_DIR}/${REL_OSARCH}/vc-scheduler ./cmd/scheduler;\
fi;

images: image_bins
images:
for name in controller-manager scheduler webhook-manager; do\
cp ${BIN_DIR}/${REL_OSARCH}/vc-$$name ./installer/dockerfile/$$name/;\
if [ ${REL_OSARCH} = linux/amd64 ];then\
docker build --no-cache -t $(IMAGE_PREFIX)-$$name:$(TAG) ./installer/dockerfile/$$name;\
elif [ ${REL_OSARCH} = linux/arm64 ];then\
docker build --no-cache -t $(IMAGE_PREFIX)-$$name-arm64:$(TAG) -f ./installer/dockerfile/$$name/Dockerfile.arm64 ./installer/dockerfile/$$name;\
else\
echo "only support x86_64 and arm64. Please build image according to your architecture";\
fi;\
rm installer/dockerfile/$$name/vc-$$name;\
docker buildx build -t "${IMAGE_PREFIX}-$$name:$(TAG)" . -f ./installer/dockerfile/$$name/Dockerfile --output=type="${BUILDX_OUTPUT_TYPE}" --platform "${DOCKER_PLATFORMS}"; \
done

webhook-manager-base-image:
if [ ${REL_OSARCH} = linux/amd64 ];then\
docker build --no-cache -t $(IMAGE_PREFIX)-webhook-manager-base:$(TAG) ./installer/dockerfile/webhook-manager/ -f ./installer/dockerfile/webhook-manager/Dockerfile.base;\
elif [ ${REL_OSARCH} = linux/arm64 ];then\
docker build --no-cache -t $(IMAGE_PREFIX)-webhook-manager-base-arm64:$(TAG) ./installer/dockerfile/webhook-manager/ -f ./installer/dockerfile/webhook-manager/Dockerfile.base.arm64;\
else\
echo "only support x86_64 and arm64. Please build webhook-manager-base-image according to your architecture";\
fi
docker buildx build -t ${IMAGE_PREFIX}-webhook-manager-base:$(TAG) . -f ./installer/dockerfile/webhook-manager/Dockerfile.base --output=type="${BUILDX_OUTPUT_TYPE}" --platform "${DOCKER_PLATFORMS}"

generate-code:
./hack/update-gencode.sh
Expand Down
6 changes: 6 additions & 0 deletions docs/development/development.md
View file
Open in desktop
Expand Up @@ -51,6 +51,12 @@ Build the containers in your local docker cache:
make images
```

To build cross-platform images:

```bash
make images DOCKER_PLATFORMS="linux/amd64,linux/arm64" BUILDX_OUTPUT_TYPE=registry IMAGE_PREFIX=[yourregistry]
```

## Building a specific docker image

If you want to make a local change and test some component, say `vc-controller-manager`, you
Expand Down
6 changes: 5 additions & 1 deletion installer/dockerfile/controller-manager/Dockerfile
View file
Open in desktop
Expand Up @@ -12,8 +12,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.

FROM golang:1.16.5 AS builder
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are three places where golang mirrors are used. If I want to update the version of go that compiles volcano, I need to update three places. Is there any room for optimization here?

WORKDIR /go/src/volcano.sh/
ADD . volcano
RUN cd volcano && make vc-controller-manager

FROM alpine:latest

ADD vc-controller-manager /vc-controller-manager
COPY --from=builder /go/src/volcano.sh/volcano/_output/bin/vc-controller-manager /vc-controller-manager
ENTRYPOINT ["/vc-controller-manager"]
18 changes: 0 additions & 18 deletions installer/dockerfile/controller-manager/Dockerfile.arm64
View file
Open in desktop

This file was deleted.

7 changes: 5 additions & 2 deletions installer/dockerfile/scheduler/Dockerfile
View file
Open in desktop
Expand Up @@ -12,8 +12,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.


FROM golang:1.16.5 AS builder
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The golang image also used here

WORKDIR /go/src/volcano.sh/
ADD . volcano
RUN cd volcano && make vc-scheduler
FROM alpine:latest

ADD vc-scheduler /vc-scheduler
COPY --from=builder /go/src/volcano.sh/volcano/_output/bin/vc-scheduler /vc-scheduler
ENTRYPOINT ["/vc-scheduler"]
18 changes: 0 additions & 18 deletions installer/dockerfile/scheduler/Dockerfile.arm64
View file
Open in desktop

This file was deleted.

9 changes: 7 additions & 2 deletions installer/dockerfile/webhook-manager/Dockerfile
View file
Open in desktop
Expand Up @@ -15,8 +15,13 @@

# The base image is created via `Dockerfile.base`, the base image is cached
# since the required packages change very rarely.
FROM golang:1.16 AS builder
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The golang image also used here

WORKDIR /go/src/volcano.sh/
ADD . volcano
RUN cd volcano && make vc-webhook-manager

FROM volcanosh/vc-webhook-manager-base:1.3.1

ADD vc-webhook-manager /vc-webhook-manager
ADD gen-admission-secret.sh /gen-admission-secret.sh
COPY --from=builder /go/src/volcano.sh/volcano/_output/bin/vc-webhook-manager /vc-webhook-manager
ADD ./installer/dockerfile/webhook-manager/gen-admission-secret.sh /gen-admission-secret.sh
ENTRYPOINT ["/vc-webhook-manager"]
19 changes: 0 additions & 19 deletions installer/dockerfile/webhook-manager/Dockerfile.arm64
View file
Open in desktop

This file was deleted.

27 changes: 0 additions & 27 deletions installer/dockerfile/webhook-manager/Dockerfile.base.arm64
View file
Open in desktop

This file was deleted.