stage1-images directory doesn't contain asc files for the aci images

[kfirufk]

Environment

rkt Version: 1.14.0
appc Version: 0.8.7
Go Version: go1.7.1
Go OS/Arch: linux/amd64

Features: -TPM +SDJOURNAL

Linux 4.7.3-coreos-r2 x86_64

NAME=CoreOS
ID=coreos
VERSION=1185.2.0
VERSION_ID=1185.2.0
BUILD_ID=2016-10-20-2326
PRETTY_NAME="CoreOS 1185.2.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"

BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

systemd 231
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN

What did you do?

configured rkt paths to overcome some deprecated configuration in kubernetes by creating
the file "/etc/rkt/paths.d/paths.json" with the content:

{
"rktKind": "paths",
"rktVersion": "v1",
"stage1-images": "/usr/lib/rkt/stage1-images"
}

What did you expect to see?

when starting rkt containers, they should start properly.

What did you see instead?

Oct 26 14:21:57 coreos-2.tux-in.com kubelet-wrapper[22576]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Oct 26 14:21:57 coreos-2.tux-in.com kubelet-wrapper[22576]: run: open /usr/lib/rkt/stage1-images/stage1-fly.aci.asc: no such file or directory

so it tires to find an asc file and it's not there.
should I manually sign the aci or am I missing something ?

[s-urbaniak]

@kfirufk thanks for the issue report. Can you do a systemctl cat of the failing service, and post it here?

[kfirufk]

sure.. it happens on calico-node and kubelet.

calico-node:

[Unit]
Description=Calico per-host agent
Requires=network-online.target
After=network-online.target

[Service]
Slice=machine.slice
PermissionsStartOnly=true
Environment=ETCD_CA_CERT_FILE=/etc/ssl/etcd/ca.pem
Environment=ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem
Environment=ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem
Environment=CALICO_DISABLE_FILE_LOGGING=true
Environment=HOSTNAME=10.79.218.2
Environment=IP=10.79.218.2
Environment=FELIX_FELIXHOSTNAME=10.79.218.2
Environment=CALICO_NETWORKING=true
Environment=NO_DEFAULT_POOLS=true
Environment=ETCD_ENDPOINTS=https://coreos-2.tux-in.com:2379,https://coreos-3.tux-in.com:2379
ExecStartPre=/bin/mkdir -p /var/run/calico
ExecStart=/usr/bin/rkt run --inherit-env --stage1-from-dir=stage1-fly.aci  --volume=var-run-calico,kind=host,source=/var/run/calico --volume=modules,kind=host,source=/lib/modules,readOnly=false --mount=volume=modules,target=/lib/modules --volume=dns,kind=host,source=/etc/resolv.conf,readOnly=true --volume=etcd-tls-certs,kind=host,source=/etc/ssl/etcd,readOnly=true --mount=volume=dns,target=/etc/resolv.conf --mount=volume=etcd-tls-certs,target=/etc/ssl/etcd --mount=volume=var-run-calico,target=/var/run/calico --trust-keys-from-https quay.io/calico/node:v0.22.0
KillMode=mixed
Restart=always
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

---

kubelet

[Service]
Environment=KUBELET_VERSION=v1.4.4_coreos.0
Environment=KUBELET_ACI=quay.io/coreos/hyperkube
Environment="RKT_OPTS=--insecure-options=image   --volume dns,kind=host,source=/etc/resolv.conf   --mount volume=dns,target=/etc/resolv.conf   --volume rkt,kind=host,source=/opt/bin/host-rkt   --mount volume=rkt,target=/usr/bin/rkt   --volume var-lib-rkt,kind=host,source=/var/lib/rkt   --mount volume=var-lib-rkt,target=/var/lib/rkt   --volume stage,kind=host,source=/tmp   --mount volume=stage,target=/tmp   --volume var-log,kind=host,source=/var/log   --mount volume=var-log,target=/var/log"
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/usr/lib/coreos/kubelet-wrapper   --kubeconfig=/etc/kubernetes/controller-kubeconfig.yaml   --register-schedulable=false   --cni-conf-dir=/etc/kubernetes/cni/net.d   --network-plugin=cni   --container-runtime=rkt   --rkt-path=/usr/bin/rkt   --allow-privileged=true   --pod-manifest-path=/etc/kubernetes/manifests   --hostname-override=10.79.218.2   --cluster_dns=10.3.0.10   --cluster_domain=cluster.local
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

[s-urbaniak]

@kfirufk thank you a lot for providing the systemd unit files, that helped a lot for debugging. /usr/lib is a symlink to /usr/lib64 in CoreOS. Furthermore only /usr/lib64/rkt/stage1-images is the trusted location for rkt on CoreOS, hence only the following config will work without any ACI checks:

{
  "rktKind": "paths",
  "rktVersion": "v1",
  "stage1-images": "/usr/lib64/rkt/stage1-images"
}

Nevertheless, I am trying to understand what you are trying to accomplish. /usr/lib64/rkt/stage1-images already is the default stage1-images directory, so why overriding it with the same value?

[kfirufk]

Hi! thanks a lot for your response. I misunderstood the rkt documentation which led me to believe that I must add this configuration for rkt to find the stage1-images. removing the file /etc/rkt/paths.d/paths.json and restarting the service did the trick. thanks again!