[runc-1.1.7-3.amzn2] Cannot run executable found relative to current directory

[jBouyoud]

What happened:

When running a pod with a private image with a ruby/rails application we got the following error message on start :

Last State: Terminated
  Reason: StartError
  Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start 
      container process: exec: "rails": cannot run executable found relative to current directory: unknown
   Exit Code: 128
   Started: Thu, 01 Jan 1970 01:00:00 +0100
   Finished: Wed, 30 Aug 2023 16:55:15 +0200

We notice some strange things :

• unknown current working dir
• Started at epoch 0

Our image define a PATH environment variable to add a relative path like ENV PATH=bin:$PATH (and WORKDIR is also correctly set on out docker image).
Adding .spec.workingDir on pod definition doesn’t change the behavior.

What you expected to happen:

To run a pod normally without any error.
The same pod definition run perfectly on v1.26.6-eks-a5565ad.

How to reproduce it (as minimally and precisely as possible):

Run the same pod definition on v1.26.7-eks-8ccc7ba  / ami-0c8f13e2e3c8de829 node.

Anything else we need to know?:

Environment:

• AWS Region: eu-west-3
• Instance Type(s): Spot - with karpenter with the following constraints :

    requirements:
    - key: topology.kubernetes.io/zone
      operator: In
      values:
      - eu-west-3a
      - eu-west-3b
      - eu-west-3c
    - key: kubernetes.io/arch
      operator: In
      values:
      - amd64
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values:
      - nano
      - micro
      - small
      - medium
      - large
      - xlarge
      - metal
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values:
      - m
      - r
      - c
    - key: karpenter.sh/capacity-type
      operator: In
      values:
      - spot
      - on-demand
    - key: kubernetes.io/os
      operator: In
      values:
      - linux

• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.5
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.26
• AMI Version: AMI Release v20230825
• Kernel (e.g. uname -a): 5.10.186-179.751.amzn2.x86_64
• Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-0f2b325398f933a81"
BUILD_TIME="Fri Aug 25 20:12:27 UTC 2023"
BUILD_KERNEL="5.10.186-179.751.amzn2.x86_64"
ARCH="x86_64"

[cartermckinnon]

This looks related to a restriction introduced in go 1.19 os/exec that executables in the PWD have to be referenced with ./: https://tip.golang.org/doc/go1.19#os-exec-path

Does that fix your issue?

[cartermckinnon]

The latest release contains a runc that was compiled with a newer version of golang (1.20.x), to address some CVE's that weren't going to be fixed in 1.18.x. That's where the behavior change is coming from.

AMI release	runc	go
v20230816	1.1.7-1.amzn2	1.18.6
v20230825	1.1.7-3.amzn2	1.20.7

[jBouyoud]

Thanks looks pretty. I'll test this and come back to you.

I'm not able to find changelog for runc and containerd for amzn2. Where do you find this information ?

[dims]

GODEBUG=execerrdot=0 is how you get back older behavior until you can fix things. There's some deeper details in https://github.com/golang/proposal/blob/master/design/56986-godebug.md as well.

[dims]

runc and containerd are Amazon Linux 2 packages, so https://alas.aws.amazon.com/alas2.html

[cartermckinnon]

go version $FILE will give you the version of go used to build $FILE as well 👍

[jBouyoud]

Hi,

Sorry, I was OoO.

I confirm that comes from GO security update.

Simply replace : ENV PATH=bin:$PATH to ENV PATH=/my/absolute/path/bin:$PATH fix the issue.

Cheers