Uploading bottlerocket images to vsphere fails

[janre]

What happened:
Uploading bottlerocket images to vsphere fails.

What you expected to happen:
New bottlerocket images should be uploaded to vsphere.

How to reproduce it (as minimally and precisely as possible):
When upgrading an existing management cluster, eksctl anywhere is not able to automatically upload the bottlerocket image to vsphere.

Anything else we need to know?:

V1	Worker node VSphereMachineConfig template is not set. Using default template.
V5	Retrier:	{"timeout": "2562047h47m16.854775807s", "backoffFactor": null}
V6	Executing command	{"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc find -json /xxx -type VirtualMachine -name bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V5	Retry execution successful	{"retries": 1, "duration": "224.185878ms"}
V2	Template not found: /xxx/vm/Templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2
V2	Template not available. Creating	{"template": "/xxx/vm/Templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V6	Executing command	{"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc library.ls eks-a-templates"}
V0	Creating template. This might take a while.
V6	Executing command	{"cmd": "/usr/bin/docker exec -i -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx eksa_1704982472982975015 govc library.info -json eks-a-templates/bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V2	Importing template from ova url	{"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova"}
V4	Importing template	{"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova", "templateName": "bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2"}
V6	Executing command	{"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=xxx -e GOVC_INSECURE=false -e GOVC_DATACENTER=xxx -e GOVC_TLS_KNOWN_HOSTS=xxx/generated/govc_known_hosts eksa_1704982472982975015 govc library.import -k -pull -n bottlerocket-v1.28.4-kubernetes-1-28-eks-10-amd64-55a1fc2 eks-a-templates https://anywhere-assets.eks.amazonaws.com/releases/bundles/54/artifacts/ova/1-28/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova"}
V9	docker	{"stderr": "govc: The import of library item 5a686415-66d7-43d0-9b93-ab7a64f1cea4 has failed. Reason: Error transferring file bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova to ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab. Reason: Error during transfer of ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab: IO error during transfer of ds:/vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f/contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-vmware-k8s-1.28-x86_64-1.15.1-264e294c_86c22c6f-777c-43fc-8c17-18daa21a926d.vmdk: Pipe closed.\n"}
V0	❌ Validation failed	{"validation": "vsphere provider validation", "error": "failed setting default values for vsphere machine configs: failed importing template into library: importing template: govc: The import of library item 5a686415-66d7-43d0-9b93-ab7a64f1cea4 has failed. Reason: Error transferring file bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64.ova to ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab. Reason: Error during transfer of ds:///vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f//contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-v1.28.4-eks-d-1-28-10-eks-a-54-amd64_86c22c6f-777c-43fc-8c17-18daa21a926d.ova?serverId=a662a392-9aea-4653-8698-d78db059ddab: IO error during transfer of ds:/vmfs/volumes/vsan:523e1d178252bed7-069a7d3eadb9e20f/contentlib-a050ed85-c354-4584-bce6-0d9fadac4e6e/5a686415-66d7-43d0-9b93-ab7a64f1cea4/bottlerocket-vmware-k8s-1.28-x86_64-1.15.1-264e294c_86c22c6f-777c-43fc-8c17-18daa21a926d.vmdk: Pipe closed.\n", "remediation": ""}

We had a support call with vmware, where an engineer found an internal error report, which says that the vmdk file of the bottlerocket image is filled with zeros at the end, which the NFC (Network File Copy) service in ESX removes as an optimization, whereby the hash of the vmdk no longer fits and the vcenter makes a rollback and deletes the file. This problem is supposed to be fixed with vsphere 8, but according to him it will no longer be patched for 7.x.

Environment:
eksctl version:
0.167.0
eksctl anywhere version:
v0.18.3
vsphere version:
7.0.3

[jaxesn]

Thanks for the report! This is a known issue that shows from time to time for certain builds of BR on certain VMware environment. We honestly havent been able to pinpoint when and why it fails.

The explanation you shared from VMWare is the first time weve heard that from them, so we can def look into that! As a workaround you can manually import the OVA into your environment and specify it in your cluster.yaml, https://anywhere.eks.amazonaws.com/docs/osmgmt/artifacts/#download-bottlerocket-node-images