rkt: docker2aci buggy tar conversion logic is broken on go1.10

[abhinavdahiya]

Issue Report

Bug

Container Linux Version

core@ip-10-0-38-189 ~ $ cat /etc/os-release 
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1722.2.0
VERSION_ID=1722.2.0
BUILD_ID=2018-03-29-0338
PRETTY_NAME="Container Linux by CoreOS 1722.2.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Amazon EC2

Actual Behavior

sudo rkt run --debug --uuid-file-save=/var/cache/kubelet-pod.uuid --volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-lib-cni,kind=host,source=/var/lib/cni --mount volume=var-lib-cni,target=/var/lib/cni --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet --mount volume=var-lib-kubelet,target=/var/lib/kubelet --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --insecure-options=image --volume coreos-etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-var-lib-docker,kind=host,source=/var/lib/docker,readOnly=false --volume coreos-var-lib-kubelet,kind=host,source=/var/lib/kubelet,readOnly=false,recursive=true --volume coreos-var-log,kind=host,source=/var/log,readOnly=false --volume coreos-os-release,kind=host,source=/usr/lib/os-release,readOnly=true --volume coreos-run,kind=host,source=/run,readOnly=false --volume coreos-lib-modules,kind=host,source=/lib/modules,readOnly=true --mount volume=coreos-etc-kubernetes,target=/etc/kubernetes --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-var-lib-docker,target=/var/lib/docker --mount volume=coreos-var-lib-kubelet,target=/var/lib/kubelet --mount volume=coreos-var-log,target=/var/log --mount volume=coreos-os-release,target=/etc/os-release --mount volume=coreos-run,target=/run --mount volume=coreos-lib-modules,target=/lib/modules --hosts-entry host --stage1-from-dir=stage1-fly.aci docker://openshift/origin:v3.10.0 --exec=/usr/bin/hyperkube -- kubelet --allow-privileged --anonymous-auth=false --cert-dir=/var/lib/kubelet/pki --client-ca-file=/etc/kubernetes/root-ca.crt --cloud-provider=aws --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/cni/bin --cni-conf-dir=/etc/kubernetes/cni/net.d --exit-on-lock-contention --kubeconfig=/etc/kubernetes/kubeconfig --lock-file=/var/run/lock/kubelet.lock --minimum-container-ttl-duration=6m0s --network-plugin=cni --node-labels=node-role.kubernetes.io/master --pod-manifest-path=/etc/kubernetes/manifests --rotate-certificates --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
image: using image from local store for url docker://openshift/origin:v3.10.0
run: 
  └─error converting docker image to ACI
    └─error generating ACI: archive/tar: cannot encode header: Format specifies USTAR; and USTAR cannot encode Linkname="rootfs/var/lib/yum/yumdb/a/6f8ba1fc41ecc5f60aac5b4b7aa28d2566c55481-audit-libs-2.7.6-3.el7-x86_64/from_repo"

Other Information

The same command seems to run on 1688.5.3

sudo rkt run --debug --uuid-file-save=/var/cache/kubelet-pod.uuid --volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-lib-cni,kind=host,source=/var/lib/cni --mount volume=var-lib-cni,target=/var/lib/cni --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet --mount volume=var-lib-kubelet,target=/var/lib/kubelet --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --insecure-options=image --volume coreos-etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-var-lib-docker,kind=host,source=/var/lib/docker,readOnly=false --volume coreos-var-lib-kubelet,kind=host,source=/var/lib/kubelet,readOnly=false,recursive=true --volume coreos-var-log,kind=host,source=/var/log,readOnly=false --volume coreos-os-release,kind=host,source=/usr/lib/os-release,readOnly=true --volume coreos-run,kind=host,source=/run,readOnly=false --volume coreos-lib-modules,kind=host,source=/lib/modules,readOnly=true --mount volume=coreos-etc-kubernetes,target=/etc/kubernetes --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-var-lib-docker,target=/var/lib/docker --mount volume=coreos-var-lib-kubelet,target=/var/lib/kubelet --mount volume=coreos-var-log,target=/var/log --mount volume=coreos-os-release,target=/etc/os-release --mount volume=coreos-run,target=/run --mount volume=coreos-lib-modules,target=/lib/modules --hosts-entry host --stage1-from-dir=stage1-fly.aci docker://openshift/origin:v3.10.0 --exec=/usr/bin/hyperkube -- kubelet --allow-privileged --anonymous-auth=false --cert-dir=/var/lib/kubelet/pki --client-ca-file=/etc/kubernetes/root-ca.crt --cloud-provider=aws --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/cni/bin --cni-conf-dir=/etc/kubernetes/cni/net.d --exit-on-lock-contention --kubeconfig=/etc/kubernetes/kubeconfig --lock-file=/var/run/lock/kubelet.lock --minimum-container-ttl-duration=6m0s --network-plugin=cni --node-labels=node-role.kubernetes.io/master --pod-manifest-path=/etc/kubernetes/manifests --rotate-certificates --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
image: using image from local store for url docker://openshift/origin:v3.10.0
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-7d2e20d13e8f25bb168ae47fe58c65a08b98c956fac9b97ea551a441e107429c
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca
stage0: Execing [/var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca/stage1/rootfs/run --debug --net=default --local-config=/etc/rkt --dns-conf-mode=resolv=default,hosts=host 0f31c779-eade-46f6-bb53-21644d6d28ca]
....

[abhinavdahiya]

/cc: @squat @bgilbert

[lucab]

This is likely a duplicate of appc/docker2aci#259

[lucab]

Additionally, this hints at a regression in go-1.10 (the one used to build current beta version) which we may temporarily workaround via downgrading the toolchain for rkt.

[lucab]

Downgrading to go-1.9 to temporarily workaround this regression.
Fix for upcoming 1758 is coreos/coreos-overlay#3175, for 1745.y is coreos/coreos-overlay#3175.