Istio CNI repair failed with "iptables": executable file not found in $PATH

[jankotulak]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

After an upgrade from v1.20.2 to 1.21.0 CNI repair functionality doesn't work anymore.

error lines from pods of istio-cni-node daemonset:

info	repair	Repairing pod...	pod=<redacted-custom-namespace>/<redacted-pod-name>
info	============= Start iptables configuration for <redacted-pod-name> =============
info	============= End iptables configuration for <redacted-pod-name> =============
error	repair	failed to setup redirection: program redirection: exec: "iptables": executable file not found in $PATH	pod=<redacted-custom-namespace>/<redacted-pod-name> netns=/host/proc/42551/ns/net
error	controllers	error handling <redacted-custom-namespace>/<redacted-pod-name>, retrying (retry count: 1): program redirection: exec: "iptables": executable file not found in $PATH	controller=repair pods
info	repair	Repairing pod...	pod=<redacted-custom-namespace>/<redacted-pod-name>
info	============= Start iptables configuration for <redacted-pod-name> =============
info	============= End iptables configuration for <redacted-pod-name> =============
error	repair	failed to setup redirection: program redirection: exec: "iptables": executable file not found in $PATH	pod=<redacted-custom-namespace>/<redacted-pod-name> netns=/host/proc/42551/ns/net
error	controllers	error handling <redacted-custom-namespace>/<redacted-pod-name>, retrying (retry count: 2): program redirection: exec: "iptables": executable file not found in $PATH	controller=repair pods
info	repair	Repairing pod...	pod=<redacted-custom-namespace>/<redacted-pod-name>
info	============= Start iptables configuration for <redacted-pod-name> =============

retry count maximum is 4 and then it starts again from number 1

Note: we use distroless images

Version

$ istioctl version
client version: 1.21.0
control plane version: 1.21.0
data plane version: 1.21.0 (9 proxies)

Additional Information

Beginning of bugreport

2024-04-02T12:12:18.503392Z     info    Running with the following config:

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }
end-time: 2024-04-02 14:12:18.3348556 +0200 CEST



2024-04-02T12:12:18.555206Z     info
Cluster endpoint: https://<redacted>.azmk8s.io:443

2024-04-02T12:12:21.228213Z     info    Done collecting cluster resource        runtime=2.6730066s
2024-04-02T12:12:21.851931Z     info    port forward completed without error
2024-04-02T12:12:23.565918Z     info    port forward completed without error
2024-04-02T12:12:24.296816Z     info    port forward completed without error
2024-04-02T12:12:24.758030Z     info    port forward completed without error
2024-04-02T12:12:24.762239Z     info    CLI version:
version.BuildInfo{Version:"1.21.0", GitRevision:"2ca3e986a683fbdb82dffcd7b2e8d02076a42468", GolangVersion:"go1.22.1", BuildStatus:"Clean", GitTag:"1.21.0"}

The following Istio control plane revisions/versions were found in the cluster:
Revision default:
&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.21.0", GitRevision:"2ca3e986a683fbdb82dffcd7b2e8d02076a42468", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.21.0"},
    },
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.21.0", GitRevision:"2ca3e986a683fbdb82dffcd7b2e8d02076a42468", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.21.0"},
    },
}

The following proxy revisions/versions were found in the cluster:
Revision default: Versions {1.21.0}

Everything else seems to be useless because of following errors during creating bugreport:

2024-04-02T12:12:25.248090Z     info    Done getting logs only for kube-system/istio-cni-node-gdjcr/    runtime=400.2067ms
2024-04-02T12:12:25.248090Z     error   open C:\Users\<REDACTED_USER>\AppData\Local\Temp\1\bug-report\bug-report\cni\istio-cni-node-gdjcr\cni.log: The system cannot find the path specified.
2024-04-02T12:12:25.248090Z     info    Done writing file for path C:\Users\<REDACTED_USER>\AppData\Local\Temp\1\bug-report\bug-report\cni\istio-cni-node-gdjcr\cni.log  runtime=0s
2024-04-02T12:12:25.248090Z     info    Done with CNI logs istio-cni-node-gdjcr

stdout:
, podExec error: error exec'ing into istio-system/istio-ingressgateway-55cc9c944c-g845c istio-proxy container: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "9ccb0a5cadcee86f9598bf39a714335ef5bec643ac82b142c052284cfe3d054c": OCI runtime exec failed: exec failed: unable to start container process: exec: "netstat": executable file not found in $PATH: unknown

, podExec error: error exec'ing into istio-system/istio-ingressgateway-55cc9c944c-g845c istio-proxy container: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "24df05081fa2d6d504e4fee7b9550b1b54835f0c11d7838e3d86dedf4bcf516f": OCI runtime exec failed: exec failed: unable to start container process: exec: "find": executable file not found in $PATH: unknown

[linsun]

@bleggett can you look at it when you have some time? Note he uses distroless image.

[bleggett]

istio-cni requires a special distroless base image with iptables binaries added (they are missing from default distroless bases that we use as the distroless base images for the other components.)

Looks like this special iptables-enabled distroless base for istio-cni did not get backported to 1.21: 531a84e

@jankotulak for now, as istio-cni always requires iptables, you will have to use the regular/non-distroless image for the istio-cni component specifically, e.g:

istioctl install --set components.cni.enabled=true --set values.global.variant=distroless --set values.cni.variant="debug"

this will use distroless base images for everything but the istio-cni component.

The next 1.21 patch release should address this such that you can use distroless for istio-cni as well and drop this workaround.

[bleggett]

This was backported to 1.21 and should be fixed in https://github.com/istio/istio/releases/tag/1.21.1

[ajaykumarmandapati]

@bleggett I believe this is still an issue for us. We are upgrading from 1.18.5 which is already EOL to the suggested 1.21.1

istioctl version                                                                                                                                      
client version: 1.21.1
control plane version: 1.21.1
data plane version: 1.18.5 (5 proxies)

however, the istio-validation container has the below error message.

2024-04-24T06:48:38.672303Z	info	Starting iptables validation. This check verifies that iptables rules are properly established for the network.
2024-04-24T06:48:38.672369Z	info	Listening on [::1]:15001
2024-04-24T06:48:38.672553Z	info	Listening on [::1]:15006
2024-04-24T06:48:43.676493Z	error	iptables validation failed; workload is not ready for Istio.
When using Istio CNI, this can occur if a pod is scheduled before the node is ready.

If installed with 'cni.repair.deletePods=true', this pod should automatically be deleted and retry.
Otherwise, this pod will need to be manually removed so that it is scheduled on a node with istio-cni running, allowing iptables rules to be established.
2024-04-24T06:48:43.676692708Z

istio-cni-node daemonset logs

# Completed on Wed Apr 24 06:48:07 2024
2024-04-24T06:48:07.423917Z	info	cni	============= End iptables configuration for echoserver-55ff5756b8-dnfkf =============
2024-04-24T06:48:21.305440Z	info	repair	Repairing pod...	pod=default/echoserver-55ff5756b8-dnfkf
2024-04-24T06:48:21.307114Z	error	controllers	error handling default/echoserver-55ff5756b8-dnfkf, retrying (retry count: 1): get netns: in host network: network id: find link for 2a05:d014:1d16:3d04:568d::3: no routes found for 2a05:d014:1d16:3d04:568d::3	controller=repair pods
2024-04-24T06:48:21.312397Z	info	repair	Repairing pod...	pod=default/echoserver-55ff5756b8-dnfkf
2024-04-24T06:48:21.312643Z	error	controllers	error handling default/echoserver-55ff5756b8-dnfkf, retrying (retry count: 2): get netns: in host network: network id: find link for 2a05:d014:1d16:3d04:568d::3: no routes found for 2a05:d014:1d16:3d04:568d::3	controller=repair pods
2024-04-24T06:48:21.322831Z	info	repair	Repairing pod...	pod=default/echoserver-55ff5756b8-dnfkf
2024-04-24T06:48:21.323075Z	error	controllers	error handling default/echoserver-55ff5756b8-dnfkf, retrying (retry count: 3): get netns: in host network: network id: find link for 2a05:d014:1d16:3d04:568d::3: no routes found for 2a05:d014:1d16:3d04:568d::3	controller=repair pods
2024-04-24T06:48:21.343280Z	info	repair	Repairing pod...	pod=default/echoserver-55ff5756b8-dnfkf
2024-04-24T06:48:21.343525Z	error	controllers	error handling default/echoserver-55ff5756b8-dnfkf, retrying (retry count: 4): get netns: in host network: network id: find link for 2a05:d014:1d16:3d04:568d::3: no routes found for 2a05:d014:1d16:3d04:568d::3	controller=repair pods

Please let me know if there is anything else I could provide?

[jankotulak]

@ajaykumarmandapati
I don't think your comment/issue is related to this bug. The bug has been resolved in v1.21.1.

@bleggett Thank you for providing a workaround as well as fix for it.

[ajaykumarmandapati]

You are right if I pull the bug-report I don't get executable file not found in $PATH: unknown anymore. Could be there is some default behavior that changed from 1.18.5 to 1.21.1

Maybe @bleggett could shed some light here? Or I can also create another issue for it. 👍🏾