Merge pull request #61241 from jpbetz/go-migrate-if-needed

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Reimplement migrate-if-needed.sh in go

The `migrate-if-needed.sh` script was already partially implemented in go (see the attachlease and rollback sub-dirs), but was still unnecessarily difficult to understand and test.  This closely reimplements the original logic but with improved code structure, error handling and testing.

Where possible, go code that was previously executed as separate binaries is now statically linked into a single 'migrate' go cobra CLI app, which is then thinly wrapped by`migrate-if-needed.sh`.

There are numerous additional improvements that need to be made, but will be submitted in future PRs.  This PR is focused on achieving parity with the pre-existing functionality and introducing some much needed test coverage, in particular HA cluster upgrade test coverage.

It appears that the `attachlease` and `rollback` go binaries are no longer needed as standalones and so I have consolidated them into the new `migrate` go binary. Other than that, this change aims to be 100% backward compatible.

```release-note
NONE
```

cluster/BUILD

@@ -16,8 +16,7 @@ filegroup(
         "//cluster/addons:all-srcs",
         "//cluster/gce:all-srcs",
         "//cluster/images/etcd-version-monitor:all-srcs",
-        "//cluster/images/etcd/attachlease:all-srcs",
-        "//cluster/images/etcd/rollback:all-srcs",
+        "//cluster/images/etcd/migrate:all-srcs",
         "//cluster/images/hyperkube:all-srcs",
         "//cluster/images/kubemark:all-srcs",
     ],

cluster/gce/config-test.sh

@@ -158,7 +158,7 @@ METADATA_AGENT_VERSION="${KUBE_METADATA_AGENT_VERSION:-0.2-0.0.19-1}"
 # Useful for scheduling heapster in large clusters with nodes of small size.
 HEAPSTER_MACHINE_TYPE="${HEAPSTER_MACHINE_TYPE:-}"
 
-# Set etcd image (e.g. k8s.gcr.io/etcd) and version (e.g. 3.1.12) if you need
+# Set etcd image (e.g. k8s.gcr.io/etcd) and version (e.g. 3.1.12-1) if you need
 # non-default version.
 ETCD_IMAGE="${TEST_ETCD_IMAGE:-}"
 ETCD_DOCKER_REPOSITORY="${TEST_ETCD_DOCKER_REPOSITORY:-}"

cluster/gce/manifests/etcd.manifest

@@ -13,7 +13,7 @@
 "containers":[
     {
     "name": "etcd-container",
-    "image": "{{ pillar.get('etcd_docker_repository', 'k8s.gcr.io/etcd') }}:{{ pillar.get('etcd_docker_tag', '3.1.12') }}",
+    "image": "{{ pillar.get('etcd_docker_repository', 'k8s.gcr.io/etcd') }}:{{ pillar.get('etcd_docker_tag', '3.1.12-1') }}",
     "resources": {
       "requests": {
         "cpu": {{ cpulimit }}
@@ -37,6 +37,12 @@
       { "name": "INITIAL_CLUSTER",
         "value": "{{ etcd_cluster }}"
       },
+      { "name": "LISTEN_PEER_URLS",
+        "value": "{{ etcd_protocol }}://{{ host_ip }}:{{ server_port }}"
+      },
+      { "name": "INITIAL_ADVERTISE_PEER_URLS",
+        "value": "{{ etcd_protocol }}://{{ hostname }}:{{ server_port }}"
+      },
       { "name": "ETCD_CREDS",
         "value": "{{ etcd_creds }}"
       }

cluster/gce/upgrade-aliases.sh

@@ -161,7 +161,7 @@ export KUBE_GCE_ENABLE_IP_ALIASES=true
 export SECONDARY_RANGE_NAME="pods-default"
 export STORAGE_BACKEND="etcd3"
 export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf"
-export ETCD_IMAGE=3.1.12
+export ETCD_IMAGE=3.1.12-1
 export ETCD_VERSION=3.1.12
 
 # Upgrade master with updated kube envs

cluster/images/etcd/Dockerfile

@@ -16,4 +16,4 @@ FROM BASEIMAGE
 
 EXPOSE 2379 2380 4001 7001
 COPY etcd* etcdctl* /usr/local/bin/
-COPY migrate-if-needed.sh start-stop-etcd.sh attachlease rollback /usr/local/bin/
+COPY migrate-if-needed.sh migrate /usr/local/bin/

cluster/images/etcd/Makefile

@@ -15,29 +15,38 @@
 # Build the etcd image
 #
 # Usage:
-# 	[TAGS=2.2.1 2.3.7 3.0.17 3.1.12] [REGISTRY=k8s.gcr.io] [ARCH=amd64] [BASEIMAGE=busybox] make (build|push)
+# 	[BUNDLED_ETCD_VERSIONS=2.2.1 2.3.7 3.0.17 3.1.12] [REGISTRY=k8s.gcr.io] [ARCH=amd64] [BASEIMAGE=busybox] make (build|push)
 
 # The image contains different etcd versions to simplify
-# upgrades. Thus be careful when removing any tag from here.
+# upgrades. Thus be careful when removing any versions from here.
 #
 # NOTE: The etcd upgrade rules are that you can upgrade only 1 minor
 # version at a time, and patch release don't matter.
 #
-# Except from etcd-$(tag) and etcdctl-$(tag) binaries, we also
+# Except from etcd-$(version) and etcdctl-$(version) binaries, we also
 # need etcd and etcdctl binaries for backward compatibility reasons.
-# That binary will be set to the last tag from $(TAGS).
-TAGS?=2.2.1 2.3.7 3.0.17 3.1.12
-REGISTRY_TAG?=3.1.12
-# ROLLBACK_REGISTRY_TAG specified the tag that REGISTRY_TAG may be rolled back to.
-ROLLBACK_REGISTRY_TAG?=3.1.12
+# That binary will be set to the last version from $(BUNDLED_ETCD_VERSIONS).
+BUNDLED_ETCD_VERSIONS?=2.2.1 2.3.7 3.0.17 3.1.12
+
+# LATEST_ETCD_VERSION identifies the most recent etcd version available.
+LATEST_ETCD_VERSION?=3.1.12
+
+# REVISION provides a version number fo this image and all it's bundled
+# artifacts. It should start at zero for each LATEST_ETCD_VERSION and increment
+# for each revision of this image at that etcd version.
+REVISION?=1
+
+# IMAGE_TAG Uniquely identifies k8s.gcr.io/etcd docker image with a tag of the form "<etcd-version>-<revision>".
+IMAGE_TAG=$(LATEST_ETCD_VERSION)-$(REVISION)
+
 ARCH?=amd64
 # Image should be pulled from k8s.gcr.io, which will auto-detect
 # region (us, eu, asia, ...) and pull from the closest.
 REGISTRY?=k8s.gcr.io
 # Images should be pushed to staging-k8s.gcr.io.
 PUSH_REGISTRY?=staging-k8s.gcr.io
-# golang version should match the golang version from https://github.com/coreos/etcd/releases for REGISTRY_TAG version of etcd.
-GOLANG_VERSION?=1.8.5
+# golang version should match the golang version from https://github.com/coreos/etcd/releases for the current ETCD_VERSION.
+GOLANG_VERSION?=1.8.7
 GOARM=7
 TEMP_DIR:=$(shell mktemp -d)
 
@@ -62,40 +71,37 @@ build:
 	# without copying the subdirectories.
 	find ./ -maxdepth 1 -type f | xargs -I {} cp {} $(TEMP_DIR)
 
-	# Compile attachlease
+	# Compile migrate
 	docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -v $(TEMP_DIR):/build -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION) \
-		/bin/bash -c "CGO_ENABLED=0 go build -o /build/attachlease k8s.io/kubernetes/cluster/images/etcd/attachlease"
-	# Compile rollback
-	docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -v $(TEMP_DIR):/build -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION) \
-		/bin/bash -c "CGO_ENABLED=0 go build -o /build/rollback k8s.io/kubernetes/cluster/images/etcd/rollback"
+		/bin/bash -c "CGO_ENABLED=0 go build -o /build/migrate k8s.io/kubernetes/cluster/images/etcd/migrate"
 
 
 ifeq ($(ARCH),amd64)
 
 	# Do not compile if we should make an image for amd64, use the official etcd binaries instead
 	# For each release create a tmp dir 'etcd_release_tmp_dir' and unpack the release tar there.
-	for tag in $(TAGS); do \
+	for version in $(BUNDLED_ETCD_VERSIONS); do \
 		etcd_release_tmp_dir=$(shell mktemp -d); \
-		curl -sSL --retry 5 https://github.com/coreos/etcd/releases/download/v$$tag/etcd-v$$tag-linux-amd64.tar.gz | tar -xz -C $$etcd_release_tmp_dir --strip-components=1; \
+		curl -sSL --retry 5 https://github.com/coreos/etcd/releases/download/v$$version/etcd-v$$version-linux-amd64.tar.gz | tar -xz -C $$etcd_release_tmp_dir --strip-components=1; \
 		cp $$etcd_release_tmp_dir/etcd $$etcd_release_tmp_dir/etcdctl $(TEMP_DIR)/; \
-		cp $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$tag; \
-		cp $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$tag; \
+		cp $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$version; \
+		cp $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$version; \
 	done
 else
 
 	# Download etcd in a golang container and cross-compile it statically
 	# For each release create a tmp dir 'etcd_release_tmp_dir' and unpack the release tar there.
-	for tag in $(TAGS); do \
+	for version in $(BUNDLED_ETCD_VERSIONS); do \
 		etcd_release_tmp_dir=$(shell mktemp -d); \
 		docker run --interactive -v $${etcd_release_tmp_dir}:/etcdbin golang:$(GOLANG_VERSION) /bin/bash -c \
 			"git clone https://github.com/coreos/etcd /go/src/github.com/coreos/etcd \
 			&& cd /go/src/github.com/coreos/etcd \
-			&& git checkout v$${tag} \
+			&& git checkout v$${version} \
 			&& GOARM=$(GOARM) GOARCH=$(ARCH) ./build \
 			&& cp -f bin/$(ARCH)/etcd* bin/etcd* /etcdbin; echo 'done'"; \
 		cp $$etcd_release_tmp_dir/etcd $$etcd_release_tmp_dir/etcdctl $(TEMP_DIR)/; \
-		cp $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$tag; \
-		cp $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$tag; \
+		cp $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$version; \
+		cp $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$version; \
 	done
 
 	# Add this ENV variable in order to workaround an unsupported arch blocker
@@ -109,139 +115,36 @@ endif
 	cd $(TEMP_DIR) && sed -i.bak 's|BASEIMAGE|$(BASEIMAGE)|g' Dockerfile
 
 	# And build the image
-	docker build --pull -t $(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) $(TEMP_DIR)
+	docker build --pull -t $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(TEMP_DIR)
 
 push: build
-	docker tag $(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) $(PUSH_REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG)
-	docker push $(PUSH_REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG)
+	docker tag $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(PUSH_REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG)
+	docker push $(PUSH_REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG)
 
 ifeq ($(ARCH),amd64)
 	# Backward compatibility. TODO: deprecate this image tag
-	docker tag $(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) $(REGISTRY)/etcd:$(REGISTRY_TAG)
-	docker push $(REGISTRY)/etcd:$(REGISTRY_TAG)
+	docker tag $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(PUSH_REGISTRY)/etcd:$(IMAGE_TAG)
+	docker push $(PUSH_REGISTRY)/etcd:$(IMAGE_TAG)
 endif
 
-ETCD2_ROLLBACK_NEW_TAG=3.0.17
-ETCD2_ROLLBACK_OLD_TAG=2.2.1
-
-# Test a rollback to etcd2 from the earliest etcd3 version.
-test-rollback-etcd2:
-	mkdir -p $(TEMP_DIR)/rollback-etcd2
-	cd $(TEMP_DIR)/rollback-etcd2
-
-	@echo "Starting $(ETCD2_ROLLBACK_NEW_TAG) etcd and writing some sample data."
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-etcd2:/var/etcd \
-		-e "TARGET_STORAGE=etcd3" \
-		-e "TARGET_VERSION=$(ETCD2_ROLLBACK_NEW_TAG)" \
-		-e "DATA_DIRECTORY=/var/etcd/data" \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'INITIAL_CLUSTER=etcd-$$(hostname)=http://localhost:2380 \
-			/usr/local/bin/migrate-if-needed.sh && \
-			source /usr/local/bin/start-stop-etcd.sh && \
-			START_STORAGE=etcd3 START_VERSION=$(ETCD2_ROLLBACK_NEW_TAG) start_etcd && \
-			ETCDCTL_API=3 /usr/local/bin/etcdctl-$(ETCD2_ROLLBACK_NEW_TAG) --endpoints http://127.0.0.1:$${ETCD_PORT} put /registry/k1 value1 && \
-			stop_etcd && \
-			[ $$(cat /var/etcd/data/version.txt) = $(ETCD2_ROLLBACK_NEW_TAG)/etcd3 ]'
-
-	@echo "Rolling back to the previous version of etcd and recording keyspace to a flat file."
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-etcd2:/var/etcd \
-		-e "TARGET_STORAGE=etcd2" \
-		-e "TARGET_VERSION=$(ETCD2_ROLLBACK_OLD_TAG)" \
-		-e "DATA_DIRECTORY=/var/etcd/data" \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'INITIAL_CLUSTER=etcd-$$(hostname)=http://localhost:2380 \
-			/usr/local/bin/migrate-if-needed.sh && \
-			source /usr/local/bin/start-stop-etcd.sh && \
-			START_STORAGE=etcd2 START_VERSION=$(ETCD2_ROLLBACK_OLD_TAG) start_etcd && \
-			/usr/local/bin/etcdctl-$(ETCD2_ROLLBACK_OLD_TAG) --endpoint 127.0.0.1:$${ETCD_PORT} get /registry/k1 > /var/etcd/keyspace.txt && \
-			stop_etcd'
-
-	@echo "Checking if rollback successfully downgraded etcd to $(ETCD2_ROLLBACK_OLD_TAG)"
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-etcd2:/var/etcd \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'[ $$(cat /var/etcd/data/version.txt) = $(ETCD2_ROLLBACK_OLD_TAG)/etcd2 ] && \
-			 grep -q value1 /var/etcd/keyspace.txt'
-
-# Test a rollback from the latest version to the previous version.
-test-rollback:
-	mkdir -p $(TEMP_DIR)/rollback-test
-	cd $(TEMP_DIR)/rollback-test
-
-	@echo "Starting $(REGISTRY_TAG) etcd and writing some sample data."
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-test:/var/etcd \
-		-e "TARGET_STORAGE=etcd3" \
-		-e "TARGET_VERSION=$(REGISTRY_TAG)" \
-		-e "DATA_DIRECTORY=/var/etcd/data" \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'INITIAL_CLUSTER=etcd-$$(hostname)=http://localhost:2380 \
-			/usr/local/bin/migrate-if-needed.sh && \
-			source /usr/local/bin/start-stop-etcd.sh && \
-			START_STORAGE=etcd3 START_VERSION=$(REGISTRY_TAG) start_etcd && \
-			ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints http://127.0.0.1:$${ETCD_PORT} put /registry/k1 value1 && \
-			stop_etcd'
-
-	@echo "Rolling back to the previous version of etcd and recording keyspace to a flat file."
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-test:/var/etcd \
-		-e "TARGET_STORAGE=etcd3" \
-		-e "TARGET_VERSION=$(ROLLBACK_REGISTRY_TAG)" \
-		-e "DATA_DIRECTORY=/var/etcd/data" \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'INITIAL_CLUSTER=etcd-$$(hostname)=http://localhost:2380 \
-			/usr/local/bin/migrate-if-needed.sh && \
-			source /usr/local/bin/start-stop-etcd.sh && \
-			START_STORAGE=etcd3 START_VERSION=$(ROLLBACK_REGISTRY_TAG) start_etcd && \
-			ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints http://127.0.0.1:$${ETCD_PORT} get --prefix / > /var/etcd/keyspace.txt && \
-			stop_etcd'
-
-	@echo "Checking if rollback successfully downgraded etcd to $(ROLLBACK_REGISTRY_TAG)"
-	docker run --tty --interactive -v $(TEMP_DIR)/rollback-test:/var/etcd \
-		$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-			'[ $$(cat /var/etcd/data/version.txt) = $(ROLLBACK_REGISTRY_TAG)/etcd3 ] && \
-			 grep -q value1 /var/etcd/keyspace.txt'
-
-# Test migrating from each supported versions to the latest version.
-test-migrate:
-	for tag in $(TAGS); do \
-		echo "Testing migration from $${tag} to $(REGISTRY_TAG)" && \
-		mkdir -p $(TEMP_DIR)/migrate-$${tag} && \
-		cd $(TEMP_DIR)/migrate-$${tag} && \
-		MAJOR_VERSION=$$(echo $${tag} | cut -c 1) && \
-		echo "Starting etcd $${tag} and writing sample data to keyspace" && \
-		docker run --tty --interactive -v $(TEMP_DIR)/migrate-$${tag}:/var/etcd \
-			-e "TARGET_STORAGE=etcd$${MAJOR_VERSION}" \
-			-e "TARGET_VERSION=$${tag}" \
-			-e "DATA_DIRECTORY=/var/etcd/data" \
-			$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-				"INITIAL_CLUSTER=etcd-\$$(hostname)=http://localhost:2380 \
-				/usr/local/bin/migrate-if-needed.sh && \
-				source /usr/local/bin/start-stop-etcd.sh && \
-				START_STORAGE=etcd$${MAJOR_VERSION} START_VERSION=$${tag} start_etcd && \
-				if [ $${MAJOR_VERSION} == 2 ]; then \
-				  /usr/local/bin/etcdctl --endpoint http://127.0.0.1:\$${ETCD_PORT} set /registry/k1 value1; \
-				else \
-				  ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints http://127.0.0.1:\$${ETCD_PORT} put /registry/k1 value1; \
-				fi && \
-				stop_etcd" && \
-		echo " Migrating from $${tag} to $(REGISTRY_TAG) and capturing keyspace" && \
-		docker run --tty --interactive -v $(TEMP_DIR)/migrate-$${tag}:/var/etcd \
-			-e "TARGET_STORAGE=etcd3" \
-			-e "TARGET_VERSION=$(REGISTRY_TAG)" \
-			-e "DATA_DIRECTORY=/var/etcd/data" \
-			$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-				'INITIAL_CLUSTER=etcd-$$(hostname)=http://localhost:2380 \
-				/usr/local/bin/migrate-if-needed.sh && \
-				source /usr/local/bin/start-stop-etcd.sh && \
-				START_STORAGE=etcd3 START_VERSION=$(REGISTRY_TAG) start_etcd && \
-				ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints http://127.0.0.1:$${ETCD_PORT} get --prefix / > /var/etcd/keyspace.txt && \
-				stop_etcd'  && \
-		echo "Checking if migrate from $${tag} successfully upgraded etcd to $(REGISTRY_TAG)" && \
-		docker run --tty --interactive -v $(TEMP_DIR)/migrate-$${tag}:/var/etcd \
-			$(REGISTRY)/etcd-$(ARCH):$(REGISTRY_TAG) /bin/sh -c \
-				'[ $$(cat /var/etcd/data/version.txt) = $(REGISTRY_TAG)/etcd3 ] && \
-				 grep -q value1 /var/etcd/keyspace.txt'; \
-	done
-
-test: test-rollback test-rollback-etcd2 test-migrate
-
+unit-test:
+	docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -e GOARCH=$(ARCH) golang:$(GOLANG_VERSION) \
+		/bin/bash -c "CGO_ENABLED=0 go test -v k8s.io/kubernetes/cluster/images/etcd/migrate"
+
+# Integration tests require both a golang build environment and all the etcd binaries from a `k8s.gcr.io/etcd` image (`/usr/local/bin/etcd-<version>`, ...).
+# Since the `k8s.gcr.io/etcd` image is for runtime only and does not have a build golang environment, we create a new docker image to run integration tests
+# with.
+build-integration-test-image: build
+	cp -r $(TEMP_DIR) $(TEMP_DIR)_integration_test
+	cp Dockerfile $(TEMP_DIR)_integration_test/Dockerfile
+	cd $(TEMP_DIR)_integration_test && sed -i.bak 's|BASEIMAGE|golang:$(GOLANG_VERSION)|g' Dockerfile
+	docker build --pull -t etcd-integration-test $(TEMP_DIR)_integration_test
+
+integration-test:
+	docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -e GOARCH=$(ARCH) etcd-integration-test \
+		/bin/bash -c "CGO_ENABLED=0 go test -tags=integration -v k8s.io/kubernetes/cluster/images/etcd/migrate -args -v 10 -logtostderr true"
+
+integration-build-test: build-integration-test-image integration-test
+test: unit-test integration-build-test
 all: build test
-.PHONY:	build push test-rollback test-rollback-etcd2 test-migrate test
+.PHONY:	build push unit-test build-integration-test-image integration-test integration-build-test test

cluster/images/etcd/README.md

@@ -1,17 +1,30 @@
-### etcd
+### k8s.gcr.io/etcd docker image
 
-This is a small etcd image used in Kubernetes setups where `etcd` is deployed as a docker image.
+Provides docker images containing etcd and etcdctl binaries for multiple etcd
+version as well as a migration operator utility for upgrading and downgrading
+etcd--it's data directory in particular--to a target version.
 
-For `amd64`, official `etcd` and `etcdctl` binaries are downloaded from Github to maintain official support.
-For other architectures, `etcd` is cross-compiled from source. Arch-specific `busybox` images serve as base images.
+#### Versioning
 
-### Upgrading and Downgrading
+Each `k8s.gcr.io/etcd` docker image is tagged with an version string of the form
+`<etcd-version>-<image-revision>`, e.g. `3.0.17-0`.  The etcd version is the
+SemVer of latest etcd version available in the image. The image revision
+distinguishes between docker images with the same lastest etcd version but
+changes (bug fixes and backward compatible improvements) to the migration
+utility bundled with the image.
 
-To upgrade to a newer etcd version, or to downgrade to the previous minor
-version, always run `/usr/local/bin/migrate-if-needed.sh` before starting the
-etcd server.
+In addition to the latest etcd version, each `k8s.gcr.io/etcd` image contains
+etcd and etcdctl binaries for older versions of etcd. These are used by the
+migration operator utility when performing downgrades and multi-step upgrades,
+but can also be used as the etcd target version.
 
-`migrate-if-needed.sh` writes a `version.txt` file to track the "current" version
+#### Usage
+
+Always run `/usr/local/bin/migrate` (or the
+`/usr/local/bin/migrate-if-needed.sh` wrapper script) before starting the etcd
+server.
+
+`migrate` writes a `version.txt` file to track the "current" version
 of etcd that was used to persist data to disk. A "target" version may also be provided
 by the `TARGET_STORAGE` (e.g. "etcd3") and `TARGET_VERSION` (e.g. "3.2.11" )
 environment variables. If the persisted version differs from the target version,
@@ -23,15 +36,30 @@ in steps to each minor version until the target version is reached.
 
 Downgrades to the previous minor version of the 3.x series and from 3.0 to 2.3.7 are supported.
 
+#### Permissions
+
+By default, `migrate` will write data directory files with default permissions
+according to the umask it is run with. When run in the published
+`k8s.gcr.io/etcd` images the default umask is 0022 which will result in 0755
+directory permissions and 0644 file permissions.
+
+#### Cross building
+
+For `amd64`, official `etcd` and `etcdctl` binaries are downloaded from Github
+to maintain official support.  For other architectures, `etcd` is cross-compiled
+from source. Arch-specific `busybox` images serve as base images.
+
 #### How to release
 
-First, run the migration and rollback tests.
+First, update `ETCD_VERSION` and `REVSION` in the `Makefile`.
+
+Next, build and test the image:
 
 ```console
 $ make build test
 ```
 
-Next, build and push the docker images for all supported architectures.
+Last, build and push the docker images for all supported architectures.
 
 ```console
 # Build for linux/amd64 (default)