init container fails with iptables: Chain already exists

[debianmaster]

init container fails with iptables: Chain already exists. message on openshift, any clue?

oc logs podname -c init gives me iptables: Chain already exists.

Version information
OpenShift Master: v3.5.5.5
Kubernetes Master: v1.5.2+43a9be4

[debianmaster]

i have following patch on deploymentConfig

[{"name":"init","image":"docker.io/istio/init:0.1","args":["-p","15001","-u","1337"],"resources":{},"imagePullPolicy":"Always","securityContext":{"capabilities":{"add":["NET_ADMIN"]}}},{"name":"enable-core-dump","image":"alpine","command":["/bin/sh"],"args":["-c","sysctl -w kernel.core_pattern=/tmp/core.%e.%p.%t \u0026\u0026 ulimit -c unlimited"],"resources":{},"imagePullPolicy":"Always","securityContext":{"privileged":true}}];

[debianmaster]

This appears to be an issue with enabling SELINUX, i had to disable selinux on all nodes to make it work.

[ldemailly]

With netadmin request it should work, or requesting specific access to iptables rather than having to disable selinux entirely?

Also why is it saying chain already exist, should be permission denied?

[bjartek]

We have the same issues trying to get it to work. For us disabling selinux is not an option. Any more information on how we can solve this will be very helpfull.

oc v3.5.5.8
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

istio 0.1.6

I followed debianmasters guide in order to try and set it up.

[debianmaster]

None as of this release. Afaik

[ayj]

The init-container needs sufficient permissions to write iptable rules for redirecting inbound/outbound traffic to the sidecar proxy. Injection requires CAP_NET_ADMIN though there seems to be potential issues with RBAC (see here) which were resolved by requesting broader privileges, i.e. "securityContext":{"privileged":true}.

With regards to SELINUX specifically, you could try some of the steps described here. It looks like you'll need some combination of updating the selinux policy and/or adding the necessary selinuxOptions to the security context of the init-container. @debianmaster, is there anyone on your side who could help with recommendations here? Once the recipe is worked out we can look into ways to rolling that support back into Istio proper alongside the existing mechanism(s).

[bjartek]

I got the init container to run with SELinux enabled. I modifed the injected init cointaner to add priviledged=true to the first init container.

"securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":true}

Your comment with mentioning requesting broader priviledges what was got my thinking of it @ayj because only the second init container was started with priviledged=true, not the first one.

If you can point me at the code I can always contribue the fix to the injected init containers. Or it might be easier for somebody with commit priviledges already to do it.

[bjartek]

It does not look like the iptables rules from https://github.com/istio/pilot/blob/master/docker/prepare_proxy.sh are present in my iptables on my nodes. I can see lots of rules from the kubelet but not this line:
iptables -t nat -N ISTIO_REDIRECT -m comment --comment "istio/redirect-common-chain"

I have done GET request towards my services but nothing shows up in grafana or servicegraph. Not so strange if the iptables rules are not present.

[rshriram]

Try to exec into your container in the pod and curl httpbin.org. See if that works. If the iptables rules are installed, it shouldn't.

On Fri, Jul 7, 2017 at 3:32 AM Bjarte S. Karlsen ***@***.***> wrote: However even though everything now is running I am not sure that the init container actually manges to modify iptables. What is it supposed to set? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH0qd3Z07z6XJfBumsfma6F2cPotWkw3ks5sLd8ggaJpZM4N7yHc> .

-- ~shriram

[ayj]

It does not look like the iptables rules from https://github.com/istio/pilot/blob/master/docker/prepare_proxy.sh are present in my iptables on my nodes.

Our init-container programs iptable rules in the pod's network namespace. The kubelet (kubeproxy) writes iptables in the node's network namespace. You'll need to exec into the pod with privileges (or equivalent via docker commands) to dump iptable rules written by prepare-proxy.sh.

@rshriram's negative test suggestion would also work.

[ayj]

If you can point me at the code I can always contribue the fix to the injected init containers. Or it might be easier for somebody with commit priviledges already to do it.

PR are always welcome and I can help with review/merges. Looks like you already found prepare_proxy.sh which is the correct place to make these kinds of fixes for now.

[bjartek]

I got it to work. I had tampered with the istio-mixer service since prometheus reported it could not scrape the 42422 port. But it looks like it works when there has been traffic.

I will send a PR to add the priviledged=true port to the injected init container in a while.

[debianmaster]

thanks @bjartek @ayj for looking into this.

[smarterclayton]

Note that adding privileged... is equivalent to turning off all security protection :)

What was the result of audit2allow when the pod was rejected with the NET_ADMIN cap but without privileged?

[bjartek]

I will try to find time to check this out @smarterclayton

[debianmaster]

@rshriram can we keep this issue open, so we can track

[ayj]

/assign @bjartek

[NeGnail]

@bjartek I have this problem now, how can I solve it quickly?

[bjartek]

You neee to add privileged=true to both init containers.

Do you have time to do what Clayton suggests above? That is run audit2allow?

[NeGnail]

@bjartek Thank you very much for your reply. I have just come into contact with this aspect. Could you please tell me where the "init containers" is? I'm a MAC.

[bjartek]

When you run istoctl inject it will modify the resource you send into it. If you are on OpenShift this is most likely your DeploymentConfig. Try to look at the top of the generered DC.

If you tell me what DOC/tutorial you are following it might be easier for me and others to help.

[NeGnail]

@bjartek I'm studying istio through this document: https://istio.io/docs/setup/consul/quick-start.html.
I have arrived at this step.：“docker-compose -f samples/bookinfo/consul/bookinfo.sidecars.yaml up -d”，but istio/proxy_init:0.4.0 status is "Exited",when I type in the 'docker ps -a' command.
This shows "iptables: Chain already exists." when I type in the 'docker logs' command.
I have been puzzling over this problem for many days. I hope you can help me.

[bjartek]

I can try to run the guide later today. What os are you on? Is selinux enabled?

[NeGnail]

@bjartek I am a MAC OS X. I'm using the docker Community Edition. thank you.

[bjartek]

Are consol_pilot_1 running? I cant get past the first step
docker-compose -f install/consul/istio.yaml up -d on my mac with docker ce. Istio pilot will not run and it needs to run in order to get the rest of the example to work.

Anyways I do not think this has anything to do with this particular issue. I would raise a new issue concerning docker-compose and istio on mac.

Have you tried it out on minikube?

[NeGnail]

@bjartek You can't start the first step, and don't know if it's because of the mistake.:
Error loading config file "/etc/istio/config/kubeconfig": read /etc/istio/config/kubeconfig: is a directory
Need to add this file to "path/install/consul/kubeconfig" on the mac.

And I've been using minikube to run istio. However, there are some problems on the network that cannot be solved and therefore no longer used.

[louiscryan]

Hey @witlessbird can you triage this ?

[dmitri-d]

@louiscryan: will take a look.

[knrc]

@louiscryan We are able to successfully deploy Istio in an SELinux environment provided we have the proxy-init container running as privileged, we are looking at alternative ways of configuring iptables to enable non-privileged deployments of Istio applications.