New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rkt: docker2aci buggy tar conversion logic is broken on go1.10 #2402

Closed
abhinavdahiya opened this Issue Apr 11, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@abhinavdahiya
Member

abhinavdahiya commented Apr 11, 2018

Issue Report

Bug

Container Linux Version

core@ip-10-0-38-189 ~ $ cat /etc/os-release 
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1722.2.0
VERSION_ID=1722.2.0
BUILD_ID=2018-03-29-0338
PRETTY_NAME="Container Linux by CoreOS 1722.2.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Amazon EC2

Actual Behavior

sudo rkt run --debug --uuid-file-save=/var/cache/kubelet-pod.uuid --volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-lib-cni,kind=host,source=/var/lib/cni --mount volume=var-lib-cni,target=/var/lib/cni --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet --mount volume=var-lib-kubelet,target=/var/lib/kubelet --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --insecure-options=image --volume coreos-etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-var-lib-docker,kind=host,source=/var/lib/docker,readOnly=false --volume coreos-var-lib-kubelet,kind=host,source=/var/lib/kubelet,readOnly=false,recursive=true --volume coreos-var-log,kind=host,source=/var/log,readOnly=false --volume coreos-os-release,kind=host,source=/usr/lib/os-release,readOnly=true --volume coreos-run,kind=host,source=/run,readOnly=false --volume coreos-lib-modules,kind=host,source=/lib/modules,readOnly=true --mount volume=coreos-etc-kubernetes,target=/etc/kubernetes --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-var-lib-docker,target=/var/lib/docker --mount volume=coreos-var-lib-kubelet,target=/var/lib/kubelet --mount volume=coreos-var-log,target=/var/log --mount volume=coreos-os-release,target=/etc/os-release --mount volume=coreos-run,target=/run --mount volume=coreos-lib-modules,target=/lib/modules --hosts-entry host --stage1-from-dir=stage1-fly.aci docker://openshift/origin:v3.10.0 --exec=/usr/bin/hyperkube -- kubelet --allow-privileged --anonymous-auth=false --cert-dir=/var/lib/kubelet/pki --client-ca-file=/etc/kubernetes/root-ca.crt --cloud-provider=aws --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/cni/bin --cni-conf-dir=/etc/kubernetes/cni/net.d --exit-on-lock-contention --kubeconfig=/etc/kubernetes/kubeconfig --lock-file=/var/run/lock/kubelet.lock --minimum-container-ttl-duration=6m0s --network-plugin=cni --node-labels=node-role.kubernetes.io/master --pod-manifest-path=/etc/kubernetes/manifests --rotate-certificates --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
image: using image from local store for url docker://openshift/origin:v3.10.0
run: 
  └─error converting docker image to ACI
    └─error generating ACI: archive/tar: cannot encode header: Format specifies USTAR; and USTAR cannot encode Linkname="rootfs/var/lib/yum/yumdb/a/6f8ba1fc41ecc5f60aac5b4b7aa28d2566c55481-audit-libs-2.7.6-3.el7-x86_64/from_repo"

Other Information

The same command seems to run on 1688.5.3

sudo rkt run --debug --uuid-file-save=/var/cache/kubelet-pod.uuid --volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-lib-cni,kind=host,source=/var/lib/cni --mount volume=var-lib-cni,target=/var/lib/cni --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet --mount volume=var-lib-kubelet,target=/var/lib/kubelet --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --insecure-options=image --volume coreos-etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-var-lib-docker,kind=host,source=/var/lib/docker,readOnly=false --volume coreos-var-lib-kubelet,kind=host,source=/var/lib/kubelet,readOnly=false,recursive=true --volume coreos-var-log,kind=host,source=/var/log,readOnly=false --volume coreos-os-release,kind=host,source=/usr/lib/os-release,readOnly=true --volume coreos-run,kind=host,source=/run,readOnly=false --volume coreos-lib-modules,kind=host,source=/lib/modules,readOnly=true --mount volume=coreos-etc-kubernetes,target=/etc/kubernetes --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-var-lib-docker,target=/var/lib/docker --mount volume=coreos-var-lib-kubelet,target=/var/lib/kubelet --mount volume=coreos-var-log,target=/var/log --mount volume=coreos-os-release,target=/etc/os-release --mount volume=coreos-run,target=/run --mount volume=coreos-lib-modules,target=/lib/modules --hosts-entry host --stage1-from-dir=stage1-fly.aci docker://openshift/origin:v3.10.0 --exec=/usr/bin/hyperkube -- kubelet --allow-privileged --anonymous-auth=false --cert-dir=/var/lib/kubelet/pki --client-ca-file=/etc/kubernetes/root-ca.crt --cloud-provider=aws --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/cni/bin --cni-conf-dir=/etc/kubernetes/cni/net.d --exit-on-lock-contention --kubeconfig=/etc/kubernetes/kubeconfig --lock-file=/var/run/lock/kubelet.lock --minimum-container-ttl-duration=6m0s --network-plugin=cni --node-labels=node-role.kubernetes.io/master --pod-manifest-path=/etc/kubernetes/manifests --rotate-certificates --register-with-taints=node-role.kubernetes.io/master=:NoSchedule
image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
image: using image from local store for url docker://openshift/origin:v3.10.0
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-7d2e20d13e8f25bb168ae47fe58c65a08b98c956fac9b97ea551a441e107429c
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca
stage0: Execing [/var/lib/rkt/pods/run/0f31c779-eade-46f6-bb53-21644d6d28ca/stage1/rootfs/run --debug --net=default --local-config=/etc/rkt --dns-conf-mode=resolv=default,hosts=host 0f31c779-eade-46f6-bb53-21644d6d28ca]
....
@abhinavdahiya

This comment has been minimized.

Member

abhinavdahiya commented Apr 11, 2018

@lucab

This comment has been minimized.

Member

lucab commented Apr 12, 2018

This is likely a duplicate of appc/docker2aci#259

@lucab

This comment has been minimized.

Member

lucab commented Apr 12, 2018

Additionally, this hints at a regression in go-1.10 (the one used to build current beta version) which we may temporarily workaround via downgrading the toolchain for rkt.

lucab added a commit to lucab/coreos-overlay that referenced this issue Apr 12, 2018

app-emulation/rkt: downgrade toolchain to go1.9
This downgrade the golang toolchain used to build rkt, keeping it at
go1.9.
This is a temporary workaround for a go-1.10 regression in `archive/tar`.

Ref: coreos/bugs#2402
Ref: appc/docker2aci#259
Ref: golang/go#24821

lucab added a commit to lucab/coreos-overlay that referenced this issue Apr 12, 2018

app-emulation/rkt: downgrade toolchain to go1.9
This downgrades the golang toolchain used to build rkt, keeping it at
go1.9.
This is a temporary workaround for a go-1.10 regression in `archive/tar`.

Ref: coreos/bugs#2402
Ref: appc/docker2aci#259
Ref: golang/go#24821

lucab added a commit to lucab/coreos-overlay that referenced this issue Apr 12, 2018

app-emulation/rkt: downgrade toolchain to go1.9
This downgrades the golang toolchain used to build rkt, keeping it at
go1.9.
This is a temporary workaround for a go-1.10 regression in `archive/tar`.

Ref: coreos/bugs#2402
Ref: appc/docker2aci#259
Ref: golang/go#24821

@lucab lucab self-assigned this Apr 12, 2018

@lucab lucab changed the title from Cannot run openshift's kubelet docker image on CoreOS 1722.2.0 to rkt: docker2aci buggy tar conversion logic is broken on go1.10 Apr 13, 2018

lucab added a commit to lucab/coreos-overlay that referenced this issue Apr 13, 2018

app-emulation/rkt: downgrade toolchain to go1.9
This downgrades the golang toolchain used to build rkt, keeping it at
go1.9.
This is a temporary workaround for a go-1.10 regression in `archive/tar`.

Ref: coreos/bugs#2402
Ref: appc/docker2aci#259
Ref: golang/go#24821
@lucab

This comment has been minimized.

Member

lucab commented Apr 16, 2018

Downgrading to go-1.9 to temporarily workaround this regression.
Fix for upcoming 1758 is coreos/coreos-overlay#3175, for 1745.y is coreos/coreos-overlay#3175.

@lucab lucab closed this May 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment