Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
8dec86c
cross-build now supports linux arm64
sgagniere Aug 25, 2022
fcb0a69
added entry to goreleaser and removed the rest of the references to t…
sgagniere Aug 25, 2022
3da0091
Some dockerfiles + build scripts, snapshot flag is temporary
sgagniere Sep 8, 2022
84ba307
Removed snapshot flag (not included, but the base image was updated t…
sgagniere Sep 15, 2022
1919e51
Modifications to release make targets
sgagniere Sep 16, 2022
e23802a
some cleanup & switched to vendoring for linux glibc amd64
sgagniere Sep 16, 2022
b7d2302
Combined the linux glibc sh files into one
sgagniere Sep 16, 2022
3b4fe85
Removed some redundant lines and rearranged others in release.mk
sgagniere Sep 16, 2022
5ca33f5
Merge from main
sgagniere Sep 16, 2022
48b8321
Repair standard_build_layout after merge
sgagniere Sep 16, 2022
7cbbccc
A comment was in a misleading spot; moved it
sgagniere Sep 16, 2022
edb474b
Oops, forgot to make generate-packaging-patch
sgagniere Sep 17, 2022
013f041
Removed redundant if else block
sgagniere Sep 19, 2022
0903953
Remove unnecessary library file
sgagniere Sep 19, 2022
bd49840
Build linux glibc versions using cross compile instead of docker images
sgagniere Sep 20, 2022
b362f61
generate packaging patch
sgagniere Sep 20, 2022
b51c8a7
Restore accidentally deleted lines
sgagniere Sep 21, 2022
e3107c1
Restore and update glibc goreleaser file
sgagniere Sep 21, 2022
1dd2e3e
missing newline
sgagniere Sep 21, 2022
2acc431
Split back into two goreleaser files to avoid goreleaser errors from …
sgagniere Sep 22, 2022
8884869
Removed disable upload flags in the goreleasers and added back the li…
sgagniere Sep 22, 2022
9969c7a
chore: minor version bump v2.26.0 [ci skip]
sgagniere Sep 22, 2022
e65ada2
Revert "chore: minor version bump v2.26.0 [ci skip]"
sgagniere Sep 22, 2022
b00269b
fixed alignment
sgagniere Sep 22, 2022
ff641cc
formatting changes
sgagniere Sep 22, 2022
e763d4f
Revert the last 11 commits (ouch)
sgagniere Sep 26, 2022
400e469
Merge branch 'main' of github.com:confluentinc/cli into linux-arm64
sgagniere Sep 26, 2022
dacddc3
Redoing formatting change that got reverted with everything else
sgagniere Sep 27, 2022
1d39e0a
update ARCHIVE_TYPES var for copying stag archives to latest
sgagniere Sep 27, 2022
bbf8485
Updated linux dockerfiles to use images from prod ecr
sgagniere Sep 27, 2022
a517cb5
added back another change that got clobbered in my reversion
sgagniere Sep 27, 2022
1f96d29
Merge branch 'main' of github.com:confluentinc/cli into linux-arm64
sgagniere Sep 29, 2022
7f393bc
Added platform flag to fix a minor annoyance where Docker will ignore…
sgagniere Sep 29, 2022
386347f
Added the same fix as the last commit, but for the amd64 image on arm…
sgagniere Sep 29, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .goreleaser-linux-glibc-arm64.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
project_name: confluent

dist: dist

before:
hooks:
# TODO: [CLI-92] we delete the semaphore cache during release to workaround an issue with semaphore and goreleaser
- rm -rf $GOPATH/pkg/mod

# NOTE: This will put all builds into the same ./dist folder. There is no way to configure goreleaser output directories per-build, only per-project.
# That means that we should probably not rely on the CI's directory layout for publishing binaries to s3 since cloud and rbac will be intermingled.
builds:
- binary: confluent
main: cmd/confluent/main.go
flags:
- -mod=vendor
ldflags:
- -s -w -X main.version={{.Env.VERSION}} -X main.commit={{.ShortCommit}} -X main.date={{.Date}} -X main.host={{.Env.HOSTNAME}}
gcflags:
- all=-trimpath={{.Env.HOME}}/git
asmflags:
- all=-trimpath={{.Env.HOME}}/git
goos:
- linux
goarch:
- arm64

release:
disable: true

archives:
- id: binary
format: binary
- id: archive
name_template: "{{ .ProjectName }}_v{{ .Version }}_{{ .Os }}_{{ .Arch }}"
format: tar.gz
format_overrides:
- goos: windows
format: zip
wrap_in_directory: "{{ .ProjectName }}"
files:
- LICENSE
- legal/**/*
2 changes: 1 addition & 1 deletion .goreleaser-linux-glibc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ builds:
- binary: confluent
main: cmd/confluent/main.go
flags:
- -mod=readonly
- -mod=vendor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind explaining why this was changed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally I did this because I had issues getting my github credentials to work on the arm64 image, but I went back and updated this amd64 file too because it feels slightly cleaner.

This way, we download all the dependencies before we copy into the image instead of downloading them in the image, and that way we don't need to copy our netrc into the cli directory and into the image.

ldflags:
- -s -w -X main.version={{.Env.VERSION}} -X main.commit={{.ShortCommit}} -X main.date={{.Date}} -X main.host={{.Env.HOSTNAME}}
gcflags:
Expand Down
30 changes: 23 additions & 7 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,27 @@ builds:
- linux
goarch:
- amd64
- binary: confluent
id: confluent-alpine-arm64
main: cmd/confluent/main.go
flags:
- -mod=readonly
- -tags=musl
ldflags:
- -s -w -X main.version={{.Env.VERSION}} -X main.commit={{.ShortCommit}} -X main.date={{.Date}} -X main.host={{.Env.HOSTNAME}}
gcflags:
- all=-trimpath={{.Env.HOME}}/git
asmflags:
- all=-trimpath={{.Env.HOME}}/git
env:
- CGO_ENABLED=1
- CC=aarch64-linux-musl-gcc
- CXX=aarch64-linux-musl-g++
- CGO_LDFLAGS=${CGO_LDFLAGS} -static
goos:
- linux
goarch:
- arm64
- binary: confluent
id: confluent-windows-amd64
main: cmd/confluent/main.go
Expand Down Expand Up @@ -98,10 +119,7 @@ builds:
goarch:
- arm64
hooks:
pre:
- cmd: make switch-librdkafka-arm64
post:
- cmd: make restore-librdkafka-amd64
- cmd: gon gon_confluent_arm64.hcl

archives:
Expand All @@ -123,15 +141,13 @@ archives:
- legal/**/*

blobs:
-
ids:
- ids:
- binary
provider: s3
bucket: confluent.cloud
region: us-west-2
folder: "{{.Env.S3FOLDER}}/binaries/{{.Version}}"
-
ids:
- ids:
- archive
provider: s3
bucket: confluent.cloud
Expand Down
26 changes: 9 additions & 17 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,18 @@ else ifneq (,$(findstring Linux,$(shell uname)))
else # build for glibc Linux
CC=gcc CXX=g++ make cli-builder
endif
else
ifneq (,$(findstring x86_64,$(shell uname -m))) # build for Darwin/amd64
make cli-builder
else # build for Darwin/arm64
make switch-librdkafka-arm64
make cli-builder || true
make restore-librdkafka-amd64
endif
else # build for Darwin
make cli-builder
endif

.PHONY: cross-build # cross-compile from Darwin/amd64 machine to Win64, Linux64 and Darwin/arm64
cross-build:
ifeq ($(GOARCH),arm64) # build for darwin/arm64.
make build-darwin-arm64
ifeq ($(GOARCH),arm64)
ifeq ($(GOOS),linux)
CGO_ENABLED=1 CC=aarch64-linux-musl-gcc CXX=aarch64-linux-musl-g++ CGO_LDFLAGS="-static" TAGS=musl make cli-builder
else # build for darwin/arm64
CGO_ENABLED=1 make cli-builder
endif
else # build for amd64 arch
ifeq ($(GOOS),windows)
CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CGO_LDFLAGS="-static" make cli-builder
Expand All @@ -38,12 +36,6 @@ else # build for amd64 arch
endif
endif

.PHONY: build-darwin-arm64
build-darwin-arm64:
make switch-librdkafka-arm64
CGO_ENABLED=1 make cli-builder || true
make restore-librdkafka-amd64

.PHONY: cli-builder
cli-builder:
@GOPRIVATE=github.com/confluentinc TAGS=$(TAGS) CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) CXX=$(CXX) CGO_LDFLAGS=$(CGO_LDFLAGS) VERSION=$(VERSION) HOSTNAME=$(HOSTNAME) goreleaser build -f .goreleaser-build.yml --rm-dist --single-target --snapshot
Expand All @@ -62,7 +54,7 @@ REF := $(shell [ -d .git ] && git rev-parse --short HEAD || echo "none")
DATE := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
HOSTNAME := $(shell id -u -n)@$(shell hostname)
RESOLVED_PATH=github.com/confluentinc/cli/cmd/confluent
RDKAFKA_VERSION = 1.8.2
RDKAFKA_VERSION = 1.9.3-RC3

S3_BUCKET_PATH=s3://confluent.cloud
S3_STAG_FOLDER_NAME=cli-release-stag
Expand Down
11 changes: 9 additions & 2 deletions build_linux_glibc.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
#!/bin/bash

cp ~/.netrc .
go mod vendor
#amd64
docker build . -f ./dockerfiles/Dockerfile_linux_glibc -t cli-linux-glibc-builder-image
docker container create --name cli-linux-glibc-builder cli-linux-glibc-builder-image
docker container cp cli-linux-glibc-builder:/go/src/github.com/confluentinc/cli/dist/. ./dist/
docker container rm cli-linux-glibc-builder
docker container rm cli-linux-glibc-builder
#arm64
docker build . -f ./dockerfiles/Dockerfile_linux_glibc_arm64 -t cli-linux-glibc-arm64-builder-image
docker container create --name cli-linux-glibc-arm64-builder cli-linux-glibc-arm64-builder-image
docker container cp cli-linux-glibc-arm64-builder:/go/src/github.com/confluentinc/cli/dist/. ./dist/
docker container rm cli-linux-glibc-arm64-builder
rm -rf vendor
32 changes: 12 additions & 20 deletions debian/patches/standard_build_layout.patch
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
--- cli/Makefile 2022-08-24 16:16:05.000000000 -0700
+++ debian/Makefile 2022-07-12 14:17:45.000000000 -0700
@@ -1,229 +1,130 @@
--- cli/Makefile 2022-09-27 16:09:05.000000000 -0700
+++ debian/Makefile 2022-09-01 10:34:50.000000000 -0700
@@ -1,221 +1,130 @@
-SHELL := /bin/bash
-ALL_SRC := $(shell find . -name "*.go" | grep -v -e vendor)
-GIT_REMOTE_NAME ?= origin
Expand All @@ -17,20 +17,18 @@
- else # build for glibc Linux
- CC=gcc CXX=g++ make cli-builder
- endif
-else
- ifneq (,$(findstring x86_64,$(shell uname -m))) # build for Darwin/amd64
- make cli-builder
- else # build for Darwin/arm64
- make switch-librdkafka-arm64
- make cli-builder || true
- make restore-librdkafka-amd64
- endif
-else # build for Darwin
- make cli-builder
-endif
-
-.PHONY: cross-build # cross-compile from Darwin/amd64 machine to Win64, Linux64 and Darwin/arm64
-cross-build:
-ifeq ($(GOARCH),arm64) # build for darwin/arm64.
- make build-darwin-arm64
-ifeq ($(GOARCH),arm64)
- ifeq ($(GOOS),linux)
- CGO_ENABLED=1 CC=aarch64-linux-musl-gcc CXX=aarch64-linux-musl-g++ CGO_LDFLAGS="-static" TAGS=musl make cli-builder
- else # build for darwin/arm64
- CGO_ENABLED=1 make cli-builder
- endif
-else # build for amd64 arch
- ifeq ($(GOOS),windows)
- CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ CGO_LDFLAGS="-static" make cli-builder
Expand All @@ -41,12 +39,6 @@
- endif
-endif
-
-.PHONY: build-darwin-arm64
-build-darwin-arm64:
- make switch-librdkafka-arm64
- CGO_ENABLED=1 make cli-builder || true
- make restore-librdkafka-amd64
-
-.PHONY: cli-builder
-cli-builder:
- @GOPRIVATE=github.com/confluentinc TAGS=$(TAGS) CGO_ENABLED=$(CGO_ENABLED) CC=$(CC) CXX=$(CXX) CGO_LDFLAGS=$(CGO_LDFLAGS) VERSION=$(VERSION) HOSTNAME=$(HOSTNAME) goreleaser build -f .goreleaser-build.yml --rm-dist --single-target --snapshot
Expand All @@ -65,7 +57,7 @@
-DATE := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
-HOSTNAME := $(shell id -u -n)@$(shell hostname)
-RESOLVED_PATH=github.com/confluentinc/cli/cmd/confluent
-RDKAFKA_VERSION = 1.8.2
-RDKAFKA_VERSION = 1.9.3-RC3
-
-S3_BUCKET_PATH=s3://confluent.cloud
-S3_STAG_FOLDER_NAME=cli-release-stag
Expand Down
6 changes: 1 addition & 5 deletions dockerfiles/Dockerfile_linux_glibc
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
FROM 037803949979.dkr.ecr.us-west-1.amazonaws.com/centos-base:latest
FROM --platform=linux/amd64 050879227952.dkr.ecr.us-west-1.amazonaws.com/confluentinc/cli-centos-base-amd64:latest

COPY . /go/src/github.com/confluentinc/cli/

COPY .netrc /root/.netrc
RUN chmod 600 /root/.netrc

RUN cd /go/src/github.com/confluentinc/cli && \
make gorelease-linux-glibc

RUN cd /go/src/github.com/confluentinc/cli && \
for file in dist/*linux*; do mv -v "$file" "${file/linux/linux}"; done ; \
for file in dist/*.txt; do mv -v "$file" "${file/checksums/checksums_linux}"; done;
9 changes: 9 additions & 0 deletions dockerfiles/Dockerfile_linux_glibc_arm64
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM --platform=linux/arm64/v8 050879227952.dkr.ecr.us-west-1.amazonaws.com/confluentinc/cli-centos-base-arm64:latest

COPY . /go/src/github.com/confluentinc/cli/

RUN cd /go/src/github.com/confluentinc/cli && \
make gorelease-linux-glibc-arm64

RUN cd /go/src/github.com/confluentinc/cli && \
for file in dist/*.txt; do mv -v "$file" "${file/checksums/checksums_linux_arm64}"; done;
18 changes: 9 additions & 9 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ require (
github.com/confluentinc/ccloud-sdk-go-v2/org v0.5.0
github.com/confluentinc/ccloud-sdk-go-v2/service-quota v0.2.0
github.com/confluentinc/cire-bucket-service/protos/bucket v0.50.0
github.com/confluentinc/confluent-kafka-go v1.8.2
github.com/confluentinc/confluent-kafka-go v1.9.3-RC3
github.com/confluentinc/countrycode v0.0.0-20211121160605-23262b771ab0
github.com/confluentinc/go-editor v0.9.0
github.com/confluentinc/go-netrc v0.0.0-20211121160620-ec37f663ea18
Expand Down Expand Up @@ -58,9 +58,9 @@ require (
github.com/havoc-io/gopass v0.0.0-20170602182606-9a121bec1ae7
github.com/iancoleman/strcase v0.2.0
github.com/imdario/mergo v0.3.12
github.com/jhump/protoreflect v1.10.2
github.com/jhump/protoreflect v1.12.0
github.com/jonboulle/clockwork v0.2.2
github.com/linkedin/goavro/v2 v2.11.0
github.com/linkedin/goavro/v2 v2.11.1
github.com/mattn/go-isatty v0.0.14
github.com/mitchellh/go-homedir v1.1.0
github.com/mitchellh/golicense v0.2.0
Expand All @@ -69,7 +69,7 @@ require (
github.com/pkg/errors v0.9.1
github.com/spf13/cobra v1.3.0
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.7.0
github.com/stretchr/testify v1.7.1
github.com/stripe/stripe-go v70.15.0+incompatible
github.com/tidwall/gjson v1.14.0
github.com/tidwall/pretty v1.2.0
Expand Down Expand Up @@ -188,7 +188,7 @@ require (
github.com/ekzhu/minhash-lsh v0.0.0-20171225071031-5c06ee8586a1 // indirect
github.com/emicklei/go-restful v2.9.6+incompatible // indirect
github.com/emirpasic/gods v1.12.0 // indirect
github.com/envoyproxy/go-control-plane v0.10.1 // indirect
github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1 // indirect
github.com/envoyproxy/protoc-gen-validate v0.6.7 // indirect
github.com/esimonov/ifshort v1.0.4 // indirect
github.com/ettle/strcase v0.1.1 // indirect
Expand Down Expand Up @@ -221,7 +221,7 @@ require (
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/golangci/check v0.0.0-20180506172741-cfe4005ccda2 // indirect
github.com/golangci/dupl v0.0.0-20180902072040-3e9179ac440a // indirect
github.com/golangci/go-misc v0.0.0-20180628070357-927a3d87b613 // indirect
Expand Down Expand Up @@ -388,9 +388,9 @@ require (
gonum.org/v1/netlib v0.0.0-20200317120129-c5a04cffd98a // indirect
google.golang.org/api v0.63.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf // indirect
google.golang.org/grpc v1.44.0 // indirect
google.golang.org/protobuf v1.27.1 // indirect
google.golang.org/genproto v0.0.0-20220503193339-ba3ae3f07e29 // indirect
google.golang.org/grpc v1.46.0 // indirect
google.golang.org/protobuf v1.28.0 // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
Expand Down
Loading