istio-init fails with "can't initialize iptables table 'nat': Permission denied

[FuzzOli87]

Is this a BUG or FEATURE REQUEST?:

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: Yes

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version - 7
kubectl version - 1.10.1
Kubernetes version - 1.9.3

Is Istio Auth enabled or not ?
Disabled

What happened:
When creating the following redis helm-chart: https://github.com/kubernetes/charts/tree/master/stable/redis

The istio-init fails with the error can't initialize iptables table 'nat': Permission denied

The StatefulSet has a defined set of securityContexts, that when they are enabled cause the error. When they are disabled, istio-init is successful.

What you expected to happen:
For the service to come up normally

How to reproduce it:
Attempt to install the helm chart in a namespace where auto-injection is ON.

[dgurindapalli]

Hi Fuzz,

Did you have solution for this issue, Even I am getting this issue after installing istio 0.8.0 with manual injection of side car into samples sleep.

[sprutner]

I also have this issue with running redis through a helm template :(

[sprutner]

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version - 0.8.0
kubectl version - 1.10.2
Kubernetes version - 1.10.3
Is Istio Auth enabled or not ?
Disabled

What happened:
When creating the following redis helm-chart: https://github.com/kubernetes/charts/tree/master/stable/redis

The istio-init fails with the error can't initialize iptables table 'nat': Permission denied

kubectl logs service-redis-master-0 -c istio-init
Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=6379,
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=

+ iptables -t nat -N ISTIO_REDIRECT
iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.

What you expected to happen:
For the service to come up normally

How to reproduce it:
Attempt to install the helm chart using a manual sidecar-injector in a namespace where auto-injection is OFF.

helm template chart --name service --set service.type=ClusterIP --namespace default > ~/service.yaml
kubectl apply -f <(istioctl kube-inject -f ~/service.yaml)

[sprutner]

Temporary workaround here: https://github.com/bitnami/bitnami-docker-redis/issues/106

  securityContext:
    runAsUser: 0

I'm running the init container as root...Works for now on master only. Getting some other errors on the slave I need to debug.

[gt-sun]

+1

# k logs prometheus-operator-6547d55767-bgwn8 -c istio-init
Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
INBOUND_CAPTURE_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=8080,
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=

+ iptables -t nat -N ISTIO_REDIRECT
iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
+ dump
+ iptables-save

[maxmilton]

Ran into this issue too when trying to helm install on GKE k8s v1.10.5. Looks like the istio-init container doesn't have enough permissions in order to change iptables.

Not sure what user it's running as by default but as a workaround setting the user to 0 a.k.a. root works (as @sprutner suggested). Example:

Assuming you've enabled sidecar auto injection:

kubectl label namespace default istio-injection=enabled

Edit the istio-sidecar-injector ConfigMap; find the text:

securityContext:\n      capabilities

and replace it with:

securityContext:\n      runAsUser: 0\n      capabilities

Rebuild your pods and they should get past the init now 😄

Fine as a workaround but I'm not super keen on this lack of security even for a simple init container.

[swistaczek]

@maxmilton this unfortunately won't help: container's runAsUser breaks non-root policy: CreateContainerConfigError

[dove-young]

I hit this problem on latest istio 1.0.4

[root@icp-311 ibm-cloud-appmgmt-prod]# kubectl logs -f icam-redis-server-696bbcc4c-v9sph -c istio-init
Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
INBOUND_CAPTURE_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=6379
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=

+ iptables -t nat -N ISTIO_REDIRECT
iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
+ dump
+ iptables-save
[root@icp-311 ibm-cloud-appmgmt-prod]#

[john-a-joyce]

Previously the Istio init-container required privileged mode. Recent work eliminates that requirement. See:
https://docs.google.com/document/d/1B5gGvdlZEfk5oAL8Qx8NdRKDYb3YlGphJoQLhwPBXYw/edit#heading=h.gjdgxs
https://github.com/istio/cni
and/or
https://deploy-preview-2902--preliminary-istio.netlify.com/docs/setup/kubernetes/istio-cni-install/

IMO - this issue should be closed.

[seanhig]

I've no idea why anyone is calling for this issue to be closed, I see no resolution apart from workarounds that involve hacking into the istio-sidecar-injector yaml. I've recently hit this issue on a kubeadm CentOS 7.5 cluster, 1.13.1 with Calico 3.4 and Istio 1.0.2.

Before closing this issue could someone provide a proper explanation? Recent work may eliminate the requirement for root level access, but it doesn't explain the error reported, or how to avoid it.

[john-a-joyce]

Did you try it with Istio master branch as described in the links I provided above? The links also have some background on why it occurs and additional links for even more background. It's no biggie for me if you want to keep this open, but it is extremely unlikely to get any further attention since from everything I can tell this issue is due to not allowing net_admin privileges as required by Istio prior to the CNI changes. Perhaps your request is the work be backported to the 1.0.X release. If so you should make that clear.

[seanhig]

Thanks @john-a-joyce, I wasn't able to get much from the links. The first link requires access I do not have. I don't know enough about the issue to be requesting a backport, so definitely not that. Just chasing down yet another K8s version clash.

I'm late to this party but hitting the same issue trying to install 1.0.1 on a 1.13.1 Calico based cluster.

I'm not sure how istio CNI fits into the picture, but it wasn't in any install guides.

Oddly I don't hit this issue installing Istio 1.0.1 to a 1.13.1 Canal based cluster. With my limited knowledge I would deduce that this is because Calico requires iptables access that flannel does not.

I've been stuck in a loop where the current release of Knative 0.2.2 seems to only work with Istio 1.0.1 or 1.0.2, but Istio pre-1.0.4 doesn't play well with Calico 3.4 on 1.13.1 (or 1.12.x)... Calico 3.4 only seems to work with Istio 1.0.4... its a sort of hell fun. It gets old.

I take it from your comments that 1.1 is the future and 1.0 is dead - but everything online leads to 1.0.x.
I've been following both the Knative and Calico online docs... depending on the combo you pick, you end up on a github issue.

It's alpha stuff, I know...

Only combo that works seems to be one where Canal is the network plugin. Then I can layer Istio and Knative on without issue...

Ok, this should be closed. Any information contained is already out of date.

[john-a-joyce]

@seanhig - I would not say 1.0 is dead, but it does require net_admin privileges without some awkward workarounds. In some environments having net_admin privileges is a non-starter, while others it isn't a big deal. If you are looking just for how to set this up on Istio without all the background the in-flight PR is your best source. istio/istio.io#2902 (review) the link in prior comment was to the rendered version which maybe you can't access.
In a nut shell. The istio sidecar model requires IPtables rules in the pod and the current 1.0 default way to do that is via net_admin privileges given to the pod itself. The Istio CNI plugin adds an Istio specific CNI to the CNI chain so these iptable rules can be done by a CNI binary that is not run in a pod context. Thus the Pod no longer needs net admin privileges. I am not an expert on which environments will not allow net_admin privileges or how to enable them in various environments so I can't really say if your specific combinations are hitting the root cause of this issue.

[rafik8]

I have the same issue when trying to install mariadb using helm chart (https://github.com/helm/charts/tree/master/stable/mariadb) on the top of OpenShift.
Istio version: 1.0.4
Kubernetes version: 1.11
Helm version: 2.11
OpenShift version: 3.11

$ kubectl logs --previous  po/dstreet-mariadb-master-0 -c istio-init
+ iptables -t nat -N ISTIO_REDIRECT
Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
Variables:
----------
PROXY_PORT=15001
INBOUND_CAPTURE_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=3306
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
iptables v1.6.0: can't initialize iptables table nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
+ dump
+ iptables-save
iptables-save v1.6.0: Cannot initialize: Permission denied (you must be root)

I am running a microservice application and we have both MySQL and MariaDB running (which are quite similar). Everything is fine for MySQL and I got the issue only with MariaDB. I am curious to know why this happens to some applications and not others. In my understand, this should happens for every deployed application if istio-init doesn't have the required permissions to perform iptables configuration.

[tuo]

same here with istio-1.0.5:

init pods always got stuck:

ᐅ kubectl version --short
Client Version: v1.12.2
Server Version: v1.10.0

ᐅ minikube version
minikube version: v0.31.0

[tuo]

Alright; problem solved; it has nothing to do with istio. The culprit seems to be the chart itself.

It happens to Postgresql chart or Redis chart which was published by Bitnami. Almost all charts it published, it by intention set RunAsUser to non-root in its chart. (I guess better security?)

I suggest you to clone the chart then comment out all parts with securityContext in all yamls.

sth like following:

Then it should work ! Now you could pack this new chart and maybe push to some chart repo and use it instead.

However in my case, my postgresql service is perfectly to run as ROOT user ; but maybe it is not case for you, you could double check with its official docs:

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

[tuo]

or you could just edit your values.yaml to set redis or postgresql's securityContext enabled to false.

redis:
master:
securityContext:
enabled: false
slave:
securityContext:
enabled: false

postgresql:
securityContext:
enabled: false

[tuo]

[rafik8]

@tuo, in my case (mariadb) this workaround was been working with OpenShift 3.10 (Kubernetes 1.10) but no longer working after upgrading to OpenShift 3.11 (Kubernetes 1.11).

[sfesfizh]

Previously the Istio init-container required privileged mode. Recent work eliminates that requirement. See:
https://docs.google.com/document/d/1B5gGvdlZEfk5oAL8Qx8NdRKDYb3YlGphJoQLhwPBXYw/edit#heading=h.gjdgxs
https://github.com/istio/cni
and/or
https://deploy-preview-2902--preliminary-istio.netlify.com/docs/setup/kubernetes/istio-cni-install/

IMO - this issue should be closed.

Yep, confirmed. This solution works and fixed problem with init container permissions.

[rafik8]

@sfesfizh, have you been able to plug Istio CNI with Istio 1.0.x? AFAIK, Istio CNI will be an experimental feature in 1.1.

[sfesfizh]

@sfesfizh, have you been able to plug Istio CNI with Istio 1.0.x? AFAIK, Istio CNI will be an experimental feature in 1.1.

Yes.
name: istio
appVersion: 1.0.4

name: cni
appVersion: 0.1.0

I've just downloaded CNI separately and moved chart as a subchart to the istio. I'm deploying it manually from my artifactory.

[rafik8]

In my case, I was been able to fix this issue by adding the security context configuration as below:

securityContext:
    enabled: false
    fsGroup: 1001
    runAsUser: 1001

I am using mariadb as an subchart (via requirement.yaml) and if I put only the configuration below I still get the init issue:

securityContext:
    enabled: false

I am still digging to know why I should add these two extra configurations (fsGroup and runAsUser) even if the security context is disabled.

[rafik8]

@sfesfizh, have you been able to plug Istio CNI with Istio 1.0.x? AFAIK, Istio CNI will be an experimental feature in 1.1.

Yes.
name: istio
appVersion: 1.0.4

name: cni
appVersion: 0.1.0

I've just downloaded CNI separately and moved chart as a subchart to the istio. I'm deploying it manually from my artifactory.

Great, nice to know that!

[sfesfizh]

In my case, I was been able to fix this issue by adding the security context configuration as below:

securityContext:
    enabled: false
    fsGroup: 1001
    runAsUser: 1001

I am using mariadb as an subchart (via requirement.yaml) and if I put only the configuration below I still get the init issue:

securityContext:
    enabled: false

I am still digging to know why I should add these two extra configurations (fsGroup and runAsUser) even if the security context is disabled.

I know but this is not an option for us. We need to keep security as well. So CNI is better solution.

[cmeyertons]

performing the securityContext: enabled = false didn't work for me unfortunately :(

I have istio in my source as a subchart of an infrastructure umbrella chart that I deploy upon cluster initialization (this allows me to save off my istio config and keep it consistent across envs).

I ended up following @maxmilton and manually editing the sidecar-injector-configmap.yaml with the following:

securityContext:
          capabilities:
            add:
            - NET_ADMIN
          privileged: true
          runAsUser: 0 # WARNING -- added to prevent https://github.com/istio/old_issues_repo/issues/316
          runAsNonRoot: false # WARNING -- added to prevent https://github.com/istio/old_issues_repo/issues/316

[sfesfizh]

performing the securityContext: enabled = false didn't work for me unfortunately :(

I have istio in my source as a subchart of an infrastructure umbrella chart that I deploy upon cluster initialization (this allows me to save off my istio config and keep it consistent across envs).

I ended up following @maxmilton and manually editing the sidecar-injector-configmap.yaml with the following:

securityContext:
          capabilities:
            add:
            - NET_ADMIN
          privileged: true
          runAsUser: 0 # WARNING -- added to prevent https://github.com/istio/old_issues_repo/issues/316
          runAsNonRoot: false # WARNING -- added to prevent https://github.com/istio/old_issues_repo/issues/316

Yes! I've fixed a configmap by adding "runAsUser: 0" and "runAsNonRoot: false" too. It works!
About CNI - it works but it breaks all namespaces by default if you didn't specify it in "exclude namespaces". It completely breaks all dynamic environment and even if you don't need istio on the whole namespace but only on specified pods.

[alexanderawitin]

I've issued PRs that fixes this particular issue for the following charts:

• PostgreSQL: [stable/postgresql] Fix "iptables: Permission Denined" error when installing on Istio-enabled cluster. helm/charts#11226
• Redis: [stable/redis] Fix "iptables: Permission Denined" error when installing on Istio-enabled cluster. helm/charts#11232