panic in istio-init #30393

vinemish · 2021-01-26T13:56:20Z

Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1611663316986592165.txt253404344
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1611663316986592165.txt253404344
iptables-restore: line 2 failed
iptables-save
# Generated by iptables-save v1.6.1 on Tue Jan 26 12:15:16 2021
*nat
:PREROUTING ACCEPT [1549:92940]
:INPUT ACCEPT [2432:145920]
:OUTPUT ACCEPT [2242:151685]
:POSTROUTING ACCEPT [3116:204125]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Tue Jan 26 12:15:16 2021
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0x3cb0a30, 0x26101e9, 0x10, 0xc001505e60, 0x2, 0x2)
        istio.io/istio/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc00133fd68, 0x3c7da01, 0x0, 0x0)
        istio.io/istio/tools/istio-iptables/pkg/cmd/run.go:554 +0x38a
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc00133fd68)
        istio.io/istio/tools/istio-iptables/pkg/cmd/run.go:561 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc00133fd68)
        istio.io/istio/tools/istio-iptables/pkg/cmd/run.go:499 +0x2b7b
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0x3c58c20, 0xc0014a2600, 0x0, 0x10)
        istio.io/istio/tools/istio-iptables/pkg/cmd/root.go:58 +0x150
github.com/spf13/cobra.(*Command).execute(0x3c58c20, 0xc00021bf00, 0x10, 0x10, 0x3c58c20, 0xc00021bf00)
        github.com/spf13/cobra@v1.0.0/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x3c58ec0, 0x0, 0x0, 0x0)
        github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
        istio.io/istio/pilot/cmd/pilot-agent/main.go:487 +0x31

[ ] Docs
[ ] Installation
[X ] Networking
[X] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

vinemish:/tmp
⚡ istioctl version --remote
client version: 1.8.2
control plane version: 1.8.2
data plane version: 1.8.2 (6 proxies)

vinemish:/tmp
⚡ kubectl version --short
Client Version: v1.17.0
Server Version: v1.18.15+IKS

Environment where the bug was observed (cloud vendor, OS, etc)
IBM cloud

Also FIRST attempted run of istio-init in the pod is always good but next attempt fails while restroing the iptables. Attaching logs of the first run.

Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1611660216882762309.txt988125463
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
iptables-restore --noflush /tmp/iptables-rules-1611660216882762309.txt988125463
Writing following contents to rules file:  /tmp/ip6tables-rules-1611660217252646072.txt032735626

ip6tables-restore --noflush /tmp/ip6tables-rules-1611660217252646072.txt032735626
iptables-save
# Generated by iptables-save v1.6.1 on Tue Jan 26 11:23:37 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Tue Jan 26 11:23:37 2021

Also I have observed that whenever packet and byte counter increases init pod crashes..

:PREROUTING ACCEPT [1549:92940]
:INPUT ACCEPT [2432:145920]
:OUTPUT ACCEPT [2242:151685]
:POSTROUTING ACCEPT [3116:204125]

If you see good run log packet and byte counters are 0. Can somebody please help me and let me know what is missing here ?

The text was updated successfully, but these errors were encountered:

howardjohn · 2021-01-26T20:34:17Z

Also FIRST attempted run of istio-init in the pod is always good but next attempt fails while restroing the iptables. Attaching logs of the first run.

Why are there multiple runs on a single pod?

vinemish · 2021-01-28T11:17:31Z

@howardjohn not sure why it's running second time but I suspect the moment PREROUTING pkt nd bytes counter increases it restarts the istio pod and starts the init pod.

hzxuzhonghu · 2021-01-29T01:32:53Z

Init container runs until successfully, there is only one possibility, it fails during pod bootstrap, and restarted. Not sure if the initcontainer start is idempotently

vinemish · 2021-02-01T17:51:15Z

Thanks @howardjohn @hzxuzhonghu any suggestions/pointers on what to check?

hzxuzhonghu · 2021-02-02T01:05:38Z

@rlenglet Could you take a look

rlenglet · 2021-02-02T14:23:31Z

Yes, this typically indicates that the init container ran more than once.
@vinemish can you please get the logs from the first run of the container? We need to know why the container failed the first time.

vinemish · 2021-02-05T13:43:25Z

Thanks much @rlenglet @hzxuzhonghu @howardjohn for your help and responses.. After going through several old git issues I found this issue kubernetes/kubernetes#67261 similar to my case.. and yes it was because of cron job clearing the init container and triggering the restart. I am marking this issue resolved and closing it.

emike922 · 2021-04-07T08:50:06Z

I think this issue should not be closed. Even despite the fact that it was a cron job responsible for clearing the init containers, it still shows that istio-iptables is NOT idempotent.

If, for whatever reason, the init containers are getting re-executed, we should make sure that they do not fail because the settings are already in effect. Even Kubernetes docs mention that init containers should be designed in such fashion:

Because init containers can be restarted, retried, or re-executed, init container code should be idempotent. In particular, code that writes to files on EmptyDirs should be prepared for the possibility that an output file already exists.
-- https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

I have observed a similar (but distinct!!) problem in an IPv6 environment. I was able to locate the problem for the IPv6 case at

istio/tools/istio-iptables/pkg/cmd/run.go

Line 370 in 4ce531f

    
           iptConfigurator.ext.RunOrFail(constants.IP, "-6", "addr", "add", "::6/128", "dev", "lo")

ip -6 addr add ::6/128 dev lo will exit(2) during reruns. (And, of course, there may still be other problems after this one, like for example the one shown by @vinemish)

@rlenglet @hzxuzhonghu @howardjohn would you be willing to reopen this case or should I create another, specific to my IPv6 issue?

Edit: IPv6 issue could be as easy as replacing ip addr add with ip addr replace. If this description is accurate, replace will perform add or change as necessary.

leosarra · 2024-04-16T13:40:40Z

Reopening this since I'm working on making istio-iptables idempotent

istio-policy-bot added area/networking area/perf and scalability feature/Multi-cluster issues related with multi-cluster support labels Jan 26, 2021

hzxuzhonghu mentioned this issue Feb 4, 2021

Istio init crash when using TPROXY #30385

Closed

vinemish closed this as completed Feb 5, 2021

sha-rath mentioned this issue Jul 5, 2021

Fixes #32145. Ensures safe re-execution of init containers. #33839

Closed

leosarra linked a pull request Apr 9, 2024 that will close this issue

Idempotency for istio-iptables apply flow #50328

Open

13 tasks

leosarra reopened this Apr 16, 2024

leosarra self-assigned this Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic in istio-init #30393

panic in istio-init #30393

vinemish commented Jan 26, 2021 •

edited by istio-policy-bot

howardjohn commented Jan 26, 2021

vinemish commented Jan 28, 2021

hzxuzhonghu commented Jan 29, 2021

vinemish commented Feb 1, 2021

hzxuzhonghu commented Feb 2, 2021

rlenglet commented Feb 2, 2021

vinemish commented Feb 5, 2021

emike922 commented Apr 7, 2021 •

edited

leosarra commented Apr 16, 2024

panic in istio-init #30393

panic in istio-init #30393

Comments

vinemish commented Jan 26, 2021 • edited by istio-policy-bot

howardjohn commented Jan 26, 2021

vinemish commented Jan 28, 2021

hzxuzhonghu commented Jan 29, 2021

vinemish commented Feb 1, 2021

hzxuzhonghu commented Feb 2, 2021

rlenglet commented Feb 2, 2021

vinemish commented Feb 5, 2021

emike922 commented Apr 7, 2021 • edited

leosarra commented Apr 16, 2024

vinemish commented Jan 26, 2021 •

edited by istio-policy-bot

emike922 commented Apr 7, 2021 •

edited