example of rebuild logic for Docker images

[errordeveloper]

I would like to share a Makefile I wrote for one JavaScript project, I think it maybe of some interest. It's not the prettiest piece of code and can be refactored, but it's works very well as is. If it is of interest, I'd be happy to take stab at a PR that implements something similar.

To be clear, one can certainly optimise Dockerfile (and .dockerignore etc) fairly well (although it's not always trivial), however copying the context take some time and it can be avoided in many cases when there are no actual changes.

IMAGE_TAG := $(shell ./tools/image-tag)  # see https://github.com/weaveworks/build-tools/blob/34d81d70a15ded472d149d0afbe41fafcf549267/image-tag
IMAGE_NAME := quay.io/weaveworks/launch-generator
UPTODATE := .uptodate/$(IMAGE_TAG)
GIT_DESCRIBE := $(shell git describe)

.PHONY: $(UPTODATE)

image: build

## When container is built, we ouput the SHA1 of the image to a `.uptodate/$(IMAGE_TAG)`
$(UPTODATE):
	docker build --tag=$(IMAGE_NAME) --tag=$(IMAGE_NAME):$(IMAGE_TAG) --build-arg=version=$(GIT_DESCRIBE) .
	docker inspect -f '{{.Id}}' $(IMAGE_NAME):$(IMAGE_TAG) > $@

## We need to quickly decide if docker build needs to run at all or not
## - always build if `$(UPTODATE)` file is missing
## - always build if `$(IMAGE_TAG)` ends with `-WIP`, i.e. there are uncommited changes
## - otherwise:
##   - if `$(IMAGE_NAME):$(IMAGE_TAG)` exists already
##     - if `$(UPTODATE)` file exists, then check contents agains SHA1 of `$(IMAGE_NAME):$(IMAGE_TAG)`
##     - else run the build, cause image can be stale with respect to the source tree
##   - else run the build anyway
build: Makefile Dockerfile src/*.js package.json
	@mkdir -p $(dir $(UPTODATE))
	@if ! [ -e $(UPTODATE) ] ; then \
	  $(MAKE) $(UPTODATE) ; \
	else \
	  if echo "$(IMAGE_NAME):$(IMAGE_TAG)" | grep -q '.*-WIP' ; then \
	    $(MAKE) $(UPTODATE) ; \
	  else \
	    if [ $$(docker images -q $(IMAGE_NAME):$(IMAGE_TAG) | wc -l) -eq 1 ] ; then \
	      if [ -e $(UPTODATE) ] ; then \
	        if ! [ $$(docker inspect -f '{{.Id}}' $(IMAGE_NAME):$(IMAGE_TAG)) = $$(cat $(UPTODATE)) ] ; then \
	          $(MAKE) $(UPTODATE) ; \
	        fi \
	      else \
	        $(MAKE) $(UPTODATE) ; \
	      fi \
	    else \
	      $(MAKE) $(UPTODATE) ; \
	    fi \
	  fi \
	fi

local: image
	./run-locally.sh $(IMAGE_NAME):$(IMAGE_TAG)

test: image run-unit-tests.sh run-integration-tests.sh .jshintrc
	./run-unit-tests.sh $(IMAGE_NAME):$(IMAGE_TAG)
	./run-integration-tests.sh $(IMAGE_NAME):$(IMAGE_TAG)

clean:
	rm -r -f .uptodate
	docker images -q $(IMAGE_NAME) | sort | uniq | xargs docker rmi -f
	docker rmi -f promtool kubectl

[r2d4]

One interesting thing that we've surfaced on the CLI is the skaffold docker deps command, which you can run on a dockerfile and it will give you a go-formattable list of dependencies. I imagined that command being used in Makefiles just like this.

https://github.com/GoogleContainerTools/skaffold/blob/master/docs/docker-commands.md

skaffold dev already works somewhat similar to this, it will capture the filesystem events and only rebuild artifacts that have those files as dependencies.

I thought if users couldn't commit to the full skaffold workflow, they could at least refactor their Makefile to use some of the logic that we use for conditional rebuilds.

[r2d4]

There is also a command for generating minimal docker context tarballs from the dependencies skaffold docker context

[errordeveloper]

@r2d4 thanks, that's very good to hear! I'll take a look at the implementation at some point, and try to reflect how it relates to what's here. I'd like to keep this open until I had a chance to do that.

[dlorenc]

Is there anything left to do for this issue?

[r2d4]

I think that this should be fixed by #661