NG.mk: A Next Generation Makefile #3505

alrs · 2017-10-01T03:11:44Z

This adds NG.mk and some -ng Dockerfiles in the images/ directory.

This ng Makefile has only been tested with the ci and version-dist targets. Most other targets have been removed to be added back individually.

This Makefile builds all of the distributable binaries in Docker containers that mount the source tree read-only. This protects the source tree from root-owned artifacts and strange side-effects from inside containers.

This Makefile uses templates to build most of the binaries, which DRYs out the build targets substantially.

This Makefile does not use .PHONY for non-phony targets.

This Makefile uses pigz instead of gzip if available, which uses all cores and is considerably faster.

make -f NG.mk -j 3 ci version-dist is concurrency-safe and allows tests to run while kubectl and required Docker containers are downloaded in the background.

This Makefile has been tested to work with GNU Make 3.8 and 4.1.

Where possible, Docker containers have been changed to use the debian:stretch-slim image, as it is the same size as the existing image but a newer release of Debian. This isn't done for the utils-builder-ng image, as socat doesn't build using the provided incantation of gcc flags on Debian stable.

During development of this branch, I've turned on full-VM support in Travis-CI and tested that make -f NG.mk version-dist will spin up build containers and create the expected artifacts. It does.

This Makefile tweaks the layout of the .build directory. This will most likely not be an issue as new upload- targets are written to the new layout, but I'm all ears if the old way of having multiple copies of each binary was necessary for some reason.

k8s-github-robot · 2017-10-01T03:12:05Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
We suggest the following additional approver: chrislovecnm

Assign the PR to them by writing /assign @chrislovecnm in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

alrs · 2017-10-01T03:13:00Z

NG.mk gets interesting on line 530. It only works if the .dockerignore in the root of the project is deleted.

alrs · 2017-10-01T03:14:51Z

make -f NG.mk linux is a good example of this experiment at work.

k8s-github-robot · 2017-10-04T15:30:38Z

@alrs PR needs rebase

alrs · 2017-10-07T17:10:00Z

I'm able to replicate my problems with Travis in a local Ubuntu 14 container, so I think my problems in Travis are down to the differences between make 3.8 and 4.1.

chrislovecnm · 2017-10-08T02:30:32Z

@alrs I had some problems with make 4.0 in the go container today. Take a look at #3556

chrislovecnm · 2017-10-11T05:09:07Z

Did you port the updates from the main makefile file to here?
Are we able to test this on a mac?
We need the bazel targets as well.
What version of make is required?
We probably want to merge and test some.

One thing we noticed is that mac docker does not like checking timestamps on a mac hard drive. Ran into that with the protokube tar ball.

chrislovecnm

First review. Probably need a couple more

chrislovecnm · 2017-10-11T05:11:45Z

images/kops-server-ng/Dockerfile

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM debian:stretch-slim


We do not use stretch slim. We use the official Debian k8s container.

I'm happy to put it back, as I mentioned in the PR description it's Debian stable instead of oldstable.

chrislovecnm · 2017-10-11T05:11:59Z

images/kops-server-ng/Dockerfile

+FROM debian:stretch-slim
+ARG KOPS_SERVER
+# TODO
+# RUN apt-get update && apt-get install --yes --reinstall lsb-base


Copy-pasted from the original kops-server image Dockerfile in images/kops-server/Dockerfile. I can't speak to what needs to be done.

chrislovecnm · 2017-10-11T05:13:26Z

images/utils-builder-ng/build-socat.sh

+
+cd /socat && \
+CFLAGS=-static LDFLAGS=-static CPPFLAGS=-static CFLAGS_APPEND=-static \
+LDFLAGS_APPEND=-static CPPFLAGS_APPEND=-static \


Wondering if we can just have bazel do this now ... not certain

alrs · 2017-10-11T05:14:16Z

n Oct 10, 2017 10:09 PM, "Chris Love" <notifications@github.com> wrote: 1. Did you port the updates from the main makefile file to here? No, do they translate? 1. Are we able to test this on a mac? I have no Mac. 1. We need the bazel targets as well. Sure. This exists in parallel specifically so that we don't need to sweat that yet. 1. What version of make is required? Tested with 3.8 and 4.1 1. We probably want to merge and test some. One thing we noticed is that mac docker does not like checking timestamps on a mac hard drive. Ran into that with the protokube tar ball. This Makefile goes to great lengths to only run make on the host, not in the container. Perhaps this fixes that issue?

chrislovecnm · 2017-10-11T06:00:38Z

Not using .phony with a mac is not working with the protokube tarball :( I a running an older version of OSX, not sure what @justinsb is running. He works off of both Linux and mac.

alrs · 2017-10-11T12:27:08Z

@chrislovecnm Could you create a gist of the output of make -f NG.mk clean && make -f NG.mk ci version-dist?

chrislovecnm · 2017-10-21T00:23:47Z

make -f NG.mk test version-dist |& tee /tmp/debug.out

First error

for t in upup/models/bindata.go federation/model/bindata.go; do if test -e $t; then rm -fv $t; fi; done
upup/models/bindata.go
federation/model/bindata.go
if test -e .build; then rm -rfv .build; fi
.build/local/go-bindata
.build/local
.build
mkdir -p .build/kops/1.8.0-alpha.1/linux/amd64
docker run -w /go/src/k8s.io/kops \
	-e "GOOS=linux" -e "GOARCH=amd64" -e "STATIC_BUILD=yes" \
	--name kops-build-linux-amd64-kops-1508544772 \
	-v `pwd`:/go/src/k8s.io/kops:ro \
	golang:1.8.3 \
	go build  \
	-ldflags "-X k8s.io/kops.Version=1.8.0-alpha.1 \
	-X k8s.io/kops.GitVersion=b334d963d " \
	-o /tmp/.build/kops/1.8.0-alpha.1/linux/amd64/kops ./cmd/kops/
federation/apply_federation.go:33:2: no buildable Go source files in /go/src/k8s.io/kops/federation/model
make: *** [.build/kops/1.8.0-alpha.1/linux/amd64/kops] Error 1

chrislovecnm · 2017-11-10T01:07:40Z

This fails:

$ S3_BUCKET=s3://clove-kops make -f NG.mk kops upload
make: *** No rule to make target `kops'.  Stop.

chrislovecnm · 2017-11-10T01:16:53Z

Latest error:

$ gmake -f NG.mk test version-dist |& tee /tmp/debug.out

redacted lots of content ... then

for t in upup/models/bindata.go federation/model/bindata.go; do if test -e $t; then rm -fv $t; fi; done
upup/models/bindata.go
federation/model/bindata.go
if test -e .build; then rm -rfv .build; fi
.build/artifacts/channels
.build/artifacts/protokube
.build/artifacts
.build/dist/images
.build/dist
.build/local/channels
.build/local/go-bindata
.build/local/kops
.build/local/protokube
.build/local
.build
mkdir -p .build/kops/1.8.0-alpha.1/linux/amd64
docker run -w /go/src/k8s.io/kops -e "GOOS=linux" -e "GOARCH=amd64" -e "STATIC_BUILD=yes" --name kops-build-kops-linux-amd64-1510276183 -v `pwd`:/go/src/k8s.io/kops:ro golang:1.8.3 go build  -ldflags "-X k8s.io/kops.Version=1.8.0-alpha.1 -X k8s.io/kops.GitVersion=35a338fda " -o /tmp/.build/kops/1.8.0-alpha.1/linux/amd64/kops ./cmd/kops/
federation/apply_federation.go:33:2: no buildable Go source files in /go/src/k8s.io/kops/federation/model
gmake: *** [NG.mk:288: .build/kops/1.8.0-alpha.1/linux/amd64/kops] Error 1

$ gmake --version
GNU Make 4.2.1
Built for x86_64-apple-darwin15.6.0
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: true

$ uname -a
Darwin clove-mbp.local 15.6.0 Darwin Kernel Version 15.6.0: Mon Jan  9 23:07:29 PST 2017; root:xnu-3248.60.11.2.1~1/RELEASE_X86_64 x86_64

Yah I need to upgrade

chrislovecnm · 2017-11-10T01:38:10Z

mkdir -p .build/kops/1.8.0-alpha.1/linux/amd64
docker run -w /go/src/k8s.io/kops -e "GOOS=linux" -e "GOARCH=amd64" -e "STATIC_BUILD=yes" --name kops-build-kops-linux-amd64-1510277483 -v `pwd`:/go/src/k8s.io/kops:ro golang:1.8.3 go build  -ldflags "-X k8s.io/kops.Version=1.8.0-alpha.1 -X k8s.io/kops.GitVersion=35a338fda " -o /tmp/.build/kops/1.8.0-alpha.1/linux/amd64/kops ./cmd/kops/
federation/apply_federation.go:33:2: no buildable Go source files in /go/src/k8s.io/kops/federation/model
make: *** [.build/kops/1.8.0-alpha.1/linux/amd64/kops] Error 1

Same error with regular make

chrislovecnm · 2017-11-10T04:24:45Z

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
ccb351a525e1        golang:1.8.3        "make -C /go/src/k..."   About an hour ago   Up About an hour                        nodeup-build-1510284565
8997b831818a        golang:1.8.3        "make -C /go/src/k..."   About an hour ago   Up About an hour                        kops-build-1510284565

I having hanging jobs ...

fejta-bot · 2018-02-20T16:37:45Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2018-03-23T05:35:54Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot · 2018-04-22T05:52:51Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 1, 2017

alrs mentioned this pull request Oct 1, 2017

makefile: add 3 PHONY targets #3483

Merged

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 4, 2017

alrs force-pushed the makefile-ng branch from 7e385df to 58106d8 Compare October 4, 2017 15:36

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 4, 2017

alrs mentioned this pull request Oct 7, 2017

mounting kubectl from the host instead to installing in protokube #3550

Merged

1 task

alrs changed the title ~~WIP: makefile-ng~~ NG.mk: A Next Generation Makefile Oct 11, 2017

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 11, 2017

chrislovecnm reviewed Oct 11, 2017

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 13, 2017

NG.mk

a2aace6

alrs force-pushed the makefile-ng branch from f0d0908 to a2aace6 Compare November 10, 2017 05:28

justinsb added this to the 1.9 milestone Nov 22, 2017

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 20, 2018

justinsb modified the milestones: 1.9.0, 1.9.1 Feb 21, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 23, 2018

k8s-ci-robot closed this Apr 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NG.mk: A Next Generation Makefile #3505

NG.mk: A Next Generation Makefile #3505

alrs commented Oct 1, 2017 •

edited

Loading

k8s-github-robot commented Oct 1, 2017

alrs commented Oct 1, 2017

alrs commented Oct 1, 2017

k8s-github-robot commented Oct 4, 2017

alrs commented Oct 7, 2017

chrislovecnm commented Oct 8, 2017

chrislovecnm commented Oct 11, 2017

chrislovecnm left a comment

chrislovecnm Oct 11, 2017

alrs Oct 11, 2017

chrislovecnm Oct 11, 2017

alrs Oct 11, 2017

chrislovecnm Oct 11, 2017

alrs commented Oct 11, 2017 via email

chrislovecnm commented Oct 11, 2017

alrs commented Oct 11, 2017

chrislovecnm commented Oct 21, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

fejta-bot commented Feb 20, 2018

fejta-bot commented Mar 23, 2018

fejta-bot commented Apr 22, 2018

NG.mk: A Next Generation Makefile #3505

NG.mk: A Next Generation Makefile #3505

Conversation

alrs commented Oct 1, 2017 • edited Loading

k8s-github-robot commented Oct 1, 2017

alrs commented Oct 1, 2017

alrs commented Oct 1, 2017

k8s-github-robot commented Oct 4, 2017

alrs commented Oct 7, 2017

chrislovecnm commented Oct 8, 2017

chrislovecnm commented Oct 11, 2017

chrislovecnm left a comment

Choose a reason for hiding this comment

chrislovecnm Oct 11, 2017

Choose a reason for hiding this comment

alrs Oct 11, 2017

Choose a reason for hiding this comment

chrislovecnm Oct 11, 2017

Choose a reason for hiding this comment

alrs Oct 11, 2017

Choose a reason for hiding this comment

chrislovecnm Oct 11, 2017

Choose a reason for hiding this comment

alrs commented Oct 11, 2017 via email

chrislovecnm commented Oct 11, 2017

alrs commented Oct 11, 2017

chrislovecnm commented Oct 21, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

chrislovecnm commented Nov 10, 2017

fejta-bot commented Feb 20, 2018

fejta-bot commented Mar 23, 2018

fejta-bot commented Apr 22, 2018

alrs commented Oct 1, 2017 •

edited

Loading