Istio debian release 1.4.2 is broken

[johscheuer]

Bug description

The /usr/local/bin/istio-start.sh script uses already the go implementation of the iptables but it is missing it in the /usr/local/bin/ dir:

sudo journalctl -fu istio
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Scheduled restart job, restart counter is at 38.
Dec 16 17:44:50 expand-vm systemd[1]: Stopped istio-sidecar: The Istio sidecar.
Dec 16 17:44:50 expand-vm systemd[1]: Started istio-sidecar: The Istio sidecar.
Dec 16 17:44:50 expand-vm istio-start.sh[4486]: /usr/local/bin/istio-start.sh: line 83: /usr/local/bin/istio-clean-iptables: No such file or directory
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Main process exited, code=exited, status=127/n/a
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Failed with result 'exit-code'.

ll /usr/local/bin/
total 130536
drwxr-xr-x  2 root root     4096 Dec 16 17:41 ./
drwxr-xr-x 10 root root     4096 Dec 11 12:40 ../
-rwxr-sr-x  1 root root 56962016 Nov 12 19:04 envoy*
-rwxr-xr-x  1 root root     1917 Nov 12 18:43 istio-clean-iptables.sh*
-rwxr-xr-x  1 root root    22421 Nov 12 18:43 istio-iptables.sh*
-rwxr-xr-x  1 root root     2189 Nov 12 18:43 istio-node-agent-start.sh*
-rwxr-xr-x  1 root root     5072 Nov 12 18:43 istio-start.sh*
-rwxr-xr-x  1 root root 20846069 Nov 12 19:05 node_agent*
-rwxr-sr-x  1 root root 55804760 Nov 12 19:05 pilot-agent*

This means the debian package for the 1.4.2 release can't be used (I check if the error still persist on the master branch)

Expected behavior

All dependencies should be installed.

Steps to reproduce the bug

Install a VM (Ubuntu or Debian) and install the debian package for 1.4.2:

curl -L https://storage.googleapis.com/istio-release/releases/1.4.2/deb/istio-sidecar.deb > istio-sidecar.deb
sudo dpkg -i istio-sidecar.deb

now simply run: sudo /usr/local/bin/istio-start.sh

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

How was Istio installed?

Environment where bug was observed (cloud vendor, OS, etc)

[johscheuer]

In the current master branch this was fixed. But the debian package still uses the iptables script and just copies it without the .sh ?

[howardjohn]

It was actually intended to still be using the shell script but without the .sh so we can easily transparently swap out the golang implementation I think

[johscheuer]

That sounds reasonable, so an easy fix for 1.4.2 will be:

sudo ln -s /usr/local/bin/istio-clean-iptables.sh /usr/local/bin/istio-clean-iptables
sudo ln -s /usr/local/bin/istio-iptables.sh /usr/local/bin/istio-iptables

[saurabh8380]

I was seeing the similar issue and even after trying the suggestion gave by @johscheuer i see the following logs when i execute sudo systemctl restart istio and run `sudo journalctl -fu istio'

Dec 18 07:34:47 istio-vm-270266498-2-853399488 istio-start.sh[29270]: # Completed on Wed Dec 18 07:34:47 2019
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Unit entered failed state.
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Failed with result 'exit-code'.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Service hold-off time over, scheduling restart.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: Stopped istio-sidecar: The Istio sidecar.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: Started istio-sidecar: The Istio sidecar.
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *mangle
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [145452:295266046]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [145452:295266046]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [154031:73520655]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [154031:73520655]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *filter
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [157057:301944455]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [164905:82757071]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *nat
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [17984:939632]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [17984:939632]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [2637:202287]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [2637:202287]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *mangle
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [176:12236]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *nat
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [81:6480]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [81:6480]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *filter
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -m state --state ESTABLISHED -j ACCEPT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -d ::1/128 -i lo -j ACCEPT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -j REJECT --reject-with icmp6-port-unreachable
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Unit entered failed state.
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Failed with result 'exit-code'.

I then have to add the following configuration ISTIO_CUSTOM_IP_TABLES=true in /var/lib/istio/envoy/sidecar.env. After restarting istio via systemctl does istio-proxy process is up and running. However, this means that I have to manage the iptables configuration manually
`

[johscheuer]

I saw the same but I didn't find some time to debug this issue hopefully I will be able to debug this during this week.

You could try to set set -x in the scripts to see where the scripts are failing (and why). I got the error on an Ubuntu 18.04 machine.

[saurabh8380]

Retried again on two VM's (ubuntu 16.04.3 LTS ) and setup istio-proxy from start. After making the sym link for the iptables scripts, it seems to work fine and i can see the iptables rule setup and istio-proxy starting up as well.

[johscheuer]

I still got the error it seems like the issue is in the clean up script (if I return with exit 0 everything comes up.

It seems like the clean script does something strange since I can see the following debug output:

+ /usr/local/bin/istio-clean-iptables
+ trap dump EXIT
+ for cmd in iptables ip6tables
+ iptables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
+ iptables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
+ iptables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
+ iptables -t nat -F ISTIO_OUTPUT
+ iptables -t nat -X ISTIO_OUTPUT
+ iptables -t nat -F ISTIO_INBOUND
+ iptables -t nat -X ISTIO_INBOUND
+ iptables -t mangle -F ISTIO_INBOUND
+ iptables -t mangle -X ISTIO_INBOUND
+ iptables -t mangle -F ISTIO_DIVERT
+ iptables -t mangle -X ISTIO_DIVERT
+ iptables -t mangle -F ISTIO_TPROXY
+ iptables -t mangle -X ISTIO_TPROXY
+ iptables -t nat -F ISTIO_REDIRECT
+ iptables -t nat -X ISTIO_REDIRECT
+ iptables -t nat -F ISTIO_IN_REDIRECT
+ iptables -t nat -X ISTIO_IN_REDIRECT
+ for cmd in iptables ip6tables
+ ip6tables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
+ ip6tables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
+ ip6tables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
+ ip6tables -t nat -F ISTIO_OUTPUT
+ ip6tables -t nat -X ISTIO_OUTPUT
+ ip6tables -t nat -F ISTIO_INBOUND
+ ip6tables -t nat -X ISTIO_INBOUND
+ ip6tables -t mangle -F ISTIO_INBOUND
+ ip6tables -t mangle -X ISTIO_INBOUND
+ ip6tables -t mangle -F ISTIO_DIVERT
+ ip6tables -t mangle -X ISTIO_DIVERT
+ ip6tables -t mangle -F ISTIO_TPROXY
+ ip6tables -t mangle -X ISTIO_TPROXY
+ ip6tables -t nat -F ISTIO_REDIRECT
+ ip6tables -t nat -X ISTIO_REDIRECT
+ ip6tables -t nat -F ISTIO_IN_REDIRECT
+ ip6tables -t nat -X ISTIO_IN_REDIRECT
+ dump
+ iptables-save
+ ip6tables-save

and after this the script stops.

On Ubuntu 18.04 I got the following iptables version:

sudo iptables -V
iptables v1.6.1

[johscheuer]

If I change the dump function in the clean script to:

function dump {
    iptables-save
    ip6tables-save
    exit $?
}

it works again 🤔 without the exit it still return 0 but the script stops for some kind of reason..

[uhinze]

Just stumbled on this as well. Tried around a bit as well and found the following chain of errors:

• many iptables and ip6tables commands fail
• errors aren't shown because all output goes to /dev/null
• also script doesn't terminate along the way as it's not set -e (by design I guess)
• last command, which is ip6tables -t nat -X ISTIO_IN_REDIRECT, also fails. When I execute this separately and with output, this gives me:

ip6tables v1.6.1: can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

and return code 3.

• trap returns the return code of this last command (compare https://stackoverflow.com/questions/41619629/exit-code-of-traps-in-bash) to the caller (istio-start.sh)
• istio-start.sh itself has set -e so it exits after this without any error really showing.

From what I can see, the solution here should be to add a exit 0 to dump() as obviously we don't care about errors in this script. What @johscheuer did in the last comment is basically the same, since ip6tables-save is 0.

[uhinze]

My current workaround is the following before calling istio-start.sh:

echo "/usr/local/bin/istio-clean-iptables.sh || true" > /usr/local/bin/istio-clean-iptables & chmod 755 /usr/local/bin/istio-clean-iptables
ln -s /usr/local/bin/istio-iptables.sh /usr/local/bin/istio-iptables

[mkoppanen]

Has this made it's way to a release? Recently upgraded from 1.3.0 and things broke

[askmeegs]

@mkoppanen my understanding is that this will be released with 1.4.4 this month