Assets mirroring fails on SHA256 enabled assets

[gjtempleton]

1. What kops version are you running? The command kops version, will display
this information.

Trying to upgrade from a v1.14.2 cluster to v1.16.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.14.9

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops update cluster --yes --phase assets

5. What happened after the commands executed?

A number of assets failed to be successfully mirrored due to the below message:

W0310 12:14:04.307921    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" calculated value "sha1:a997fd4809d96df84cf438efeff9bdd1bc8c69f2"

The assets failing to publish were as follows:

• nodeup
• kubelet
• protokube.tar.gz
• kops
• utils.tar.gz
• kubectl

6. What did you expect to happen?

The artefacts to be successfully mirrored to the private S3 bucket.

From following the code through I think the issue was introduced in 1.15-alpha by #7468 as we now will check against sha256 hashes if they exist, but compare them against SHA1 hashes always.

**7. Please provide your cluster manifest. Execute

Believe this is N/A so leaving out for now.

**8. Please run the commands with most verbose logging by adding the -v 10 flag.

I0310 12:13:43.864736    1498 executor.go:143] No progress made, sleeping before retrying 7 failed task(s)
I0310 12:13:53.864975    1498 executor.go:103] Tasks: 220 done / 227 total; 7 can run
I0310 12:13:53.865024    1498 executor.go:176] Executing task "CopyFile/https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl": *assettasks.CopyFile {"Name":"https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl","SourceFile":"https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl","SHA":"d2a31e87c5f6deced4ba8899f9c465e54822f0cd146f32ea83cb1daafa5d9c4f","Lifecycle":"Sync"}
I0310 12:13:53.865051    1498 executor.go:176] Executing task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz": *assettasks.CopyFile {"Name":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz","SourceFile":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz","SHA":"e473a232468db76a63663055ea6ed1e12f5d7e0357285cef080a78cb0361dee0","Lifecycle":"Sync"}
I0310 12:13:53.865122    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256
I0310 12:13:53.865120    1498 executor.go:176] Executing task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz": *assettasks.CopyFile {"Name":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz","SourceFile":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz","SHA":"f0474049e5f5e98f60f83961827d5468f2f707a5f7446e77535e267b7312200e","Lifecycle":"Sync"}
I0310 12:13:53.865162    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256
I0310 12:13:53.865206    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256
I0310 12:13:53.865182    1498 executor.go:176] Executing task "CopyFile/https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet": *assettasks.CopyFile {"Name":"https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet","SourceFile":"https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet","SHA":"01c35041ffbae9a13baa4b7790066a9274ef2dd32a8fd5b3d7c03944fe9c1af7","Lifecycle":"Sync"}
I0310 12:13:53.865060    1498 executor.go:176] Executing task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops": *assettasks.CopyFile {"Name":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops","SourceFile":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops","SHA":"2ec48b66ea18859407a9b9121b6acbedc9182db3e95578ee879c333a53f69748","Lifecycle":"Sync"}
I0310 12:13:53.865111    1498 executor.go:176] Executing task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup": *assettasks.CopyFile {"Name":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup","SourceFile":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup","SHA":"06630fce88dc39b52a9373c3fa006e3b9f74b50d7c72be3be025bb90669d8f14","Lifecycle":"Sync"}
I0310 12:13:53.865303    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256
I0310 12:13:53.865280    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256
I0310 12:13:53.865099    1498 executor.go:176] Executing task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops": *assettasks.CopyFile {"Name":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops","SourceFile":"https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops","TargetFile":"https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops","SHA":"dca9c93161d50d67f316e6e45ee86c2870bfeff0c5f07a10f8ae5bb847976a80","Lifecycle":"Sync"}
I0310 12:13:53.865312    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256
I0310 12:13:53.865377    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256
I0310 12:13:53.875870    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.885333    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.886458    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.890416    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.891264    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.892106    1498 context.go:247] retrying after error file does not exist
I0310 12:13:53.896861    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.376125    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256
I0310 12:13:54.385582    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256
I0310 12:13:54.386680    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/release/v1.14.9/bin/linux/amd64/kubectl.sha256
I0310 12:13:54.390615    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256
I0310 12:13:54.391009    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.391474    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256
I0310 12:13:54.392307    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256
I0310 12:13:54.394967    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.397058    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256
I0310 12:13:54.399714    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.405636    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.414912    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.415427    1498 context.go:247] retrying after error file does not exist
I0310 12:13:54.415902    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.391225    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256
I0310 12:13:55.395185    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256
I0310 12:13:55.395904    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.399891    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256
I0310 12:13:55.403996    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.405840    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256
I0310 12:13:55.410339    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.415117    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256
I0310 12:13:55.415598    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256
I0310 12:13:55.416073    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256
I0310 12:13:55.420813    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.424477    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.425519    1498 context.go:247] retrying after error file does not exist
I0310 12:13:55.437650    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.396240    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256
I0310 12:13:57.399569    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.404237    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256
I0310 12:13:57.408488    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.410558    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256
I0310 12:13:57.414848    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.421018    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256
I0310 12:13:57.424389    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.424682    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256
I0310 12:13:57.425695    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256
I0310 12:13:57.428648    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.429933    1498 context.go:247] retrying after error file does not exist
I0310 12:13:57.437942    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256
I0310 12:13:57.442436    1498 context.go:247] retrying after error file does not exist
I0310 12:14:01.399834    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256
I0310 12:14:01.404078    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.404098    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.404169    1498 copyfile.go:130] copying bits from "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz"
I0310 12:14:01.404189    1498 context.go:179] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz
I0310 12:14:01.408764    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256
I0310 12:14:01.414491    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.414506    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.414551    1498 copyfile.go:130] copying bits from "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet"
I0310 12:14:01.414572    1498 context.go:179] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet
I0310 12:14:01.415096    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256
I0310 12:14:01.419244    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.419259    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.419306    1498 copyfile.go:130] copying bits from "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops"
I0310 12:14:01.419325    1498 context.go:179] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops
I0310 12:14:01.424626    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256
I0310 12:14:01.428447    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.428462    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.428504    1498 copyfile.go:130] copying bits from "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup"
I0310 12:14:01.428523    1498 context.go:179] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup
I0310 12:14:01.428839    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256
I0310 12:14:01.430186    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256
I0310 12:14:01.433025    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.433041    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.433083    1498 copyfile.go:130] copying bits from "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl"
I0310 12:14:01.433103    1498 context.go:179] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl
I0310 12:14:01.434215    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.434230    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.434280    1498 copyfile.go:130] copying bits from "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz"
I0310 12:14:01.434302    1498 context.go:179] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz
I0310 12:14:01.442673    1498 context.go:179] Performing HTTP request: GET https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256
I0310 12:14:01.446919    1498 context.go:251] hit maximum retries 5 with error file does not exist
I0310 12:14:01.446934    1498 copyfile.go:78] unable to download: "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256", assuming target file is not present, and if not present may not be an error: file does not exist
I0310 12:14:01.446979    1498 copyfile.go:130] copying bits from "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops"
I0310 12:14:01.447000    1498 context.go:179] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops
W0310 12:14:04.307870    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/nodeup.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/nodeup" calculated value "sha1:517f7b5e821b8d49424e94d0d35f33adcf17bdc5"
W0310 12:14:04.307904    1498 executor.go:128] error running task "CopyFile/https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet" (6m48s remaining to succeed): unable to transfer "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet": the sha value in "s3://<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet.sha256" does not match "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubelet" calculated value "sha1:f84c48d5368d6f1d687b530b673d2d093c12fe56"
W0310 12:14:04.307921    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/images/protokube.tar.gz.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/images/protokube.tar.gz" calculated value "sha1:a997fd4809d96df84cf438efeff9bdd1bc8c69f2"
W0310 12:14:04.307940    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/kops.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/kops" calculated value "sha1:95295b67781a57d9c1a578e1dd1a67c5052a6f03"
W0310 12:14:04.307958    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/darwin/amd64/kops.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/darwin/amd64/kops" calculated value "sha1:c8044a249f4373f941135f72c9f1a82292442806"
W0310 12:14:04.307974    1498 executor.go:128] error running task "CopyFile/https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz" (6m48s remaining to succeed): unable to transfer "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz": the sha value in "s3://<BUCKET_NAME_REDACTED>/kops/1.16.0/linux/amd64/utils.tar.gz.sha256" does not match "https://kubeupv2.s3.amazonaws.com/kops/1.16.0/linux/amd64/utils.tar.gz" calculated value "sha1:13f380f3c5f2df41b419e1eadaf50f886021a3fc"
W0310 12:14:04.307991    1498 executor.go:128] error running task "CopyFile/https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl" (6m48s remaining to succeed): unable to transfer "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl" to "https://s3-eu-west-1.amazonaws.com/<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl": the sha value in "s3://<BUCKET_NAME_REDACTED>/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl.sha256" does not match "https://storage.googleapis.com/kubernetes-release/release/v1.14.9/bin/linux/amd64/kubectl" calculated value "sha1:90579134a6ae9693cb39fb2c6150de1b36e8af76"
I0310 12:14:04.308006    1498 executor.go:143] No progress made, sleeping before retrying 7 failed task(s)

9. Anything else do we need to know?

Believe this will be resolved by #8694 - testing now.

[gjtempleton]

Resolved.