Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istio debian release 1.4.2 is broken #19615

Closed
johscheuer opened this issue Dec 16, 2019 · 13 comments
Closed

Istio debian release 1.4.2 is broken #19615

johscheuer opened this issue Dec 16, 2019 · 13 comments

Comments

@johscheuer
Copy link
Member

@johscheuer johscheuer commented Dec 16, 2019

Bug description

The /usr/local/bin/istio-start.sh script uses already the go implementation of the iptables but it is missing it in the /usr/local/bin/ dir:

sudo journalctl -fu istio
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Scheduled restart job, restart counter is at 38.
Dec 16 17:44:50 expand-vm systemd[1]: Stopped istio-sidecar: The Istio sidecar.
Dec 16 17:44:50 expand-vm systemd[1]: Started istio-sidecar: The Istio sidecar.
Dec 16 17:44:50 expand-vm istio-start.sh[4486]: /usr/local/bin/istio-start.sh: line 83: /usr/local/bin/istio-clean-iptables: No such file or directory
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Main process exited, code=exited, status=127/n/a
Dec 16 17:44:50 expand-vm systemd[1]: istio.service: Failed with result 'exit-code'.
ll /usr/local/bin/
total 130536
drwxr-xr-x  2 root root     4096 Dec 16 17:41 ./
drwxr-xr-x 10 root root     4096 Dec 11 12:40 ../
-rwxr-sr-x  1 root root 56962016 Nov 12 19:04 envoy*
-rwxr-xr-x  1 root root     1917 Nov 12 18:43 istio-clean-iptables.sh*
-rwxr-xr-x  1 root root    22421 Nov 12 18:43 istio-iptables.sh*
-rwxr-xr-x  1 root root     2189 Nov 12 18:43 istio-node-agent-start.sh*
-rwxr-xr-x  1 root root     5072 Nov 12 18:43 istio-start.sh*
-rwxr-xr-x  1 root root 20846069 Nov 12 19:05 node_agent*
-rwxr-sr-x  1 root root 55804760 Nov 12 19:05 pilot-agent*

This means the debian package for the 1.4.2 release can't be used (I check if the error still persist on the master branch)

Expected behavior

All dependencies should be installed.

Steps to reproduce the bug

Install a VM (Ubuntu or Debian) and install the debian package for 1.4.2:

curl -L https://storage.googleapis.com/istio-release/releases/1.4.2/deb/istio-sidecar.deb > istio-sidecar.deb
sudo dpkg -i istio-sidecar.deb

now simply run: sudo /usr/local/bin/istio-start.sh

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

How was Istio installed?

Environment where bug was observed (cloud vendor, OS, etc)

@johscheuer

This comment has been minimized.

Copy link
Member Author

@johscheuer johscheuer commented Dec 16, 2019

In the current master branch this was fixed. But the debian package still uses the iptables script and just copies it without the .sh ?

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 16, 2019

It was actually intended to still be using the shell script but without the .sh so we can easily transparently swap out the golang implementation I think

@johscheuer

This comment has been minimized.

Copy link
Member Author

@johscheuer johscheuer commented Dec 17, 2019

That sounds reasonable, so an easy fix for 1.4.2 will be:

sudo ln -s /usr/local/bin/istio-clean-iptables.sh /usr/local/bin/istio-clean-iptables
sudo ln -s /usr/local/bin/istio-iptables.sh /usr/local/bin/istio-iptables
@saurabh8380

This comment has been minimized.

Copy link

@saurabh8380 saurabh8380 commented Dec 18, 2019

I was seeing the similar issue and even after trying the suggestion gave by @johscheuer i see the following logs when i execute sudo systemctl restart istio and run `sudo journalctl -fu istio'

Dec 18 07:34:47 istio-vm-270266498-2-853399488 istio-start.sh[29270]: # Completed on Wed Dec 18 07:34:47 2019
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Unit entered failed state.
Dec 18 07:34:47 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Failed with result 'exit-code'.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Service hold-off time over, scheduling restart.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: Stopped istio-sidecar: The Istio sidecar.
Dec 18 07:34:57 istio-vm-270266498-2-853399488 systemd[1]: Started istio-sidecar: The Istio sidecar.
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *mangle
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [145452:295266046]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [145452:295266046]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [154031:73520655]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [154031:73520655]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *filter
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [157057:301944455]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [164905:82757071]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by iptables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *nat
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [17984:939632]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [17984:939632]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [2637:202287]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [2637:202287]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *mangle
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [176:12236]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *nat
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :PREROUTING ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [81:6480]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :POSTROUTING ACCEPT [81:6480]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Generated by ip6tables-save v1.6.0 on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: *filter
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :INPUT ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :FORWARD ACCEPT [0:0]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: :OUTPUT ACCEPT [162:11340]
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -m state --state ESTABLISHED -j ACCEPT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -d ::1/128 -i lo -j ACCEPT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: -A INPUT -j REJECT --reject-with icmp6-port-unreachable
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: COMMIT
Dec 18 07:34:58 istio-vm-270266498-2-853399488 istio-start.sh[29337]: # Completed on Wed Dec 18 07:34:58 2019
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Unit entered failed state.
Dec 18 07:34:58 istio-vm-270266498-2-853399488 systemd[1]: istio.service: Failed with result 'exit-code'.

I then have to add the following configuration ISTIO_CUSTOM_IP_TABLES=true in /var/lib/istio/envoy/sidecar.env. After restarting istio via systemctl does istio-proxy process is up and running. However, this means that I have to manage the iptables configuration manually
`

@johscheuer

This comment has been minimized.

Copy link
Member Author

@johscheuer johscheuer commented Dec 18, 2019

I saw the same but I didn't find some time to debug this issue hopefully I will be able to debug this during this week.

You could try to set set -x in the scripts to see where the scripts are failing (and why). I got the error on an Ubuntu 18.04 machine.

@saurabh8380

This comment has been minimized.

Copy link

@saurabh8380 saurabh8380 commented Dec 18, 2019

Retried again on two VM's (ubuntu 16.04.3 LTS ) and setup istio-proxy from start. After making the sym link for the iptables scripts, it seems to work fine and i can see the iptables rule setup and istio-proxy starting up as well.

@johscheuer

This comment has been minimized.

Copy link
Member Author

@johscheuer johscheuer commented Dec 18, 2019

I still got the error it seems like the issue is in the clean up script (if I return with exit 0 everything comes up.

It seems like the clean script does something strange since I can see the following debug output:

+ /usr/local/bin/istio-clean-iptables
+ trap dump EXIT
+ for cmd in iptables ip6tables
+ iptables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
+ iptables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
+ iptables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
+ iptables -t nat -F ISTIO_OUTPUT
+ iptables -t nat -X ISTIO_OUTPUT
+ iptables -t nat -F ISTIO_INBOUND
+ iptables -t nat -X ISTIO_INBOUND
+ iptables -t mangle -F ISTIO_INBOUND
+ iptables -t mangle -X ISTIO_INBOUND
+ iptables -t mangle -F ISTIO_DIVERT
+ iptables -t mangle -X ISTIO_DIVERT
+ iptables -t mangle -F ISTIO_TPROXY
+ iptables -t mangle -X ISTIO_TPROXY
+ iptables -t nat -F ISTIO_REDIRECT
+ iptables -t nat -X ISTIO_REDIRECT
+ iptables -t nat -F ISTIO_IN_REDIRECT
+ iptables -t nat -X ISTIO_IN_REDIRECT
+ for cmd in iptables ip6tables
+ ip6tables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
+ ip6tables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
+ ip6tables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
+ ip6tables -t nat -F ISTIO_OUTPUT
+ ip6tables -t nat -X ISTIO_OUTPUT
+ ip6tables -t nat -F ISTIO_INBOUND
+ ip6tables -t nat -X ISTIO_INBOUND
+ ip6tables -t mangle -F ISTIO_INBOUND
+ ip6tables -t mangle -X ISTIO_INBOUND
+ ip6tables -t mangle -F ISTIO_DIVERT
+ ip6tables -t mangle -X ISTIO_DIVERT
+ ip6tables -t mangle -F ISTIO_TPROXY
+ ip6tables -t mangle -X ISTIO_TPROXY
+ ip6tables -t nat -F ISTIO_REDIRECT
+ ip6tables -t nat -X ISTIO_REDIRECT
+ ip6tables -t nat -F ISTIO_IN_REDIRECT
+ ip6tables -t nat -X ISTIO_IN_REDIRECT
+ dump
+ iptables-save
+ ip6tables-save

and after this the script stops.

On Ubuntu 18.04 I got the following iptables version:

sudo iptables -V
iptables v1.6.1
@johscheuer

This comment has been minimized.

Copy link
Member Author

@johscheuer johscheuer commented Dec 18, 2019

If I change the dump function in the clean script to:

function dump {
    iptables-save
    ip6tables-save
    exit $?
}

it works again 🤔 without the exit it still return 0 but the script stops for some kind of reason..

@uhinze

This comment has been minimized.

Copy link

@uhinze uhinze commented Jan 8, 2020

Just stumbled on this as well. Tried around a bit as well and found the following chain of errors:

  • many iptables and ip6tables commands fail
  • errors aren't shown because all output goes to /dev/null
  • also script doesn't terminate along the way as it's not set -e (by design I guess)
  • last command, which is ip6tables -t nat -X ISTIO_IN_REDIRECT, also fails. When I execute this separately and with output, this gives me:
ip6tables v1.6.1: can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

and return code 3.

From what I can see, the solution here should be to add a exit 0 to dump() as obviously we don't care about errors in this script. What @johscheuer did in the last comment is basically the same, since ip6tables-save is 0.

@uhinze

This comment has been minimized.

Copy link

@uhinze uhinze commented Jan 14, 2020

My current workaround is the following before calling istio-start.sh:

echo "/usr/local/bin/istio-clean-iptables.sh || true" > /usr/local/bin/istio-clean-iptables & chmod 755 /usr/local/bin/istio-clean-iptables
ln -s /usr/local/bin/istio-iptables.sh /usr/local/bin/istio-iptables
@mkoppanen

This comment has been minimized.

Copy link

@mkoppanen mkoppanen commented Jan 27, 2020

Has this made it's way to a release? Recently upgraded from 1.3.0 and things broke

@askmeegs

This comment has been minimized.

Copy link

@askmeegs askmeegs commented Feb 2, 2020

@mkoppanen my understanding is that this will be released with 1.4.4 this month

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Feb 14, 2020

My understanding is this is fixed. If not, please let me know

note: on the 1.6 roadmap is VM hardening so hopefully we don't have these issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.