istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting #23279

gnz00 · 2020-04-26T06:05:19Z

Unable to start on the latest debian due to Istio dependency on iptables.
Bug description

Expected behavior

Steps to reproduce the bug

docker run -it debian:stable
$ apt-get update && apt-get install iptables curl
$ curl -L https://storage.googleapis.com/istio-release/releases/1.5.2/deb/istio-sidecar.deb > istio-sidecar.deb
$ dpkg -i istio-sidecar.deb
$ istio-start.sh

iptables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
iptables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
iptables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
iptables -t nat -F ISTIO_OUTPUT
iptables -t nat -X ISTIO_OUTPUT
iptables -t nat -F ISTIO_INBOUND
iptables -t nat -X ISTIO_INBOUND
iptables -t mangle -F ISTIO_INBOUND
iptables -t mangle -X ISTIO_INBOUND
iptables -t mangle -F ISTIO_DIVERT
iptables -t mangle -X ISTIO_DIVERT
iptables -t mangle -F ISTIO_TPROXY
iptables -t mangle -X ISTIO_TPROXY
iptables -t nat -F ISTIO_REDIRECT
iptables -t nat -X ISTIO_REDIRECT
iptables -t nat -F ISTIO_IN_REDIRECT
iptables -t nat -X ISTIO_IN_REDIRECT
ip6tables -t nat -D PREROUTING -p tcp -j ISTIO_INBOUND
ip6tables -t mangle -D PREROUTING -p tcp -j ISTIO_INBOUND
ip6tables -t nat -D OUTPUT -p tcp -j ISTIO_OUTPUT
ip6tables -t nat -F ISTIO_OUTPUT
ip6tables -t nat -X ISTIO_OUTPUT
ip6tables -t nat -F ISTIO_INBOUND
ip6tables -t nat -X ISTIO_INBOUND
ip6tables -t mangle -F ISTIO_INBOUND
ip6tables -t mangle -X ISTIO_INBOUND
ip6tables -t mangle -F ISTIO_DIVERT
ip6tables -t mangle -X ISTIO_DIVERT
ip6tables -t mangle -F ISTIO_TPROXY
ip6tables -t mangle -X ISTIO_TPROXY
ip6tables -t nat -F ISTIO_REDIRECT
ip6tables -t nat -X ISTIO_REDIRECT
ip6tables -t nat -F ISTIO_IN_REDIRECT
ip6tables -t nat -X ISTIO_IN_REDIRECT
iptables-save
ip6tables-save
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=101,0
PROXY_GID=101,0
INBOUND_INTERCEPTION_MODE=
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1587880711852484700.txt607378426
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 101 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 101 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 101 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 0 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 0 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 0 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 101 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 101 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 101 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 0 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 0 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 0 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1587880711852484700.txt607378426
iptables-restore v1.8.2 (nf_tables): unknown option "--to-port"
Error occurred at line: 5
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables-save
panic: exit status 2

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739ca, 0x10, 0xc000092ba0, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000117d30, 0x443001, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:473 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000117d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:480 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000117d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:427 +0x24e2
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xd819c0, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc0000a8010, 0x0, 0x0, 0xd5c740, 0xc0000a8010)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00007a058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.5.2

How was Istio installed?
Manually installed the debian package, see above.

Environment where bug was observed (cloud vendor, OS, etc)
Docker on OSX running Debian buster

The text was updated successfully, but these errors were encountered:

howardjohn · 2020-04-26T14:21:58Z

@rlenglet @abhide @lambdai what is the fix here? Do we need to support nftables or use some subset of the iptables command that nftables supports or something? Not super familiar here but this will likely become more common as time goes on

rlenglet · 2020-04-26T15:39:26Z

@gnz00 what is your Linux kernel version?

rlenglet · 2020-04-26T15:45:38Z

I think the vanilla iptables command is too forgiving. We're passing a --to-port flag whereas we should be passing --to-ports (with a s) according to the manpage. nftables's iptables command may be more strict.

@gnz00 could you do a quick check that this works if you replace --to-port with --to-ports in the commands here and a few other places in the same file?

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes istio#23279

rlenglet · 2020-04-27T04:11:00Z

Keeping this open until I cherry-pick #23289 into release-1.5.

rlenglet · 2020-04-27T04:55:47Z

Changing to --to-ports didn't fix this issue. It seems like nftables's iptables command just doesn't support the --to-ports flag: #23289 (comment).
This is weird because nftables does support the equivalent of REDIRECT.

rlenglet · 2020-04-27T05:14:47Z

Note that this issue is not blocking usage on Debian Buster. iptables can be switched back to not using nftables:

$ update-alternatives --set iptables /usr/sbin/iptables-legacy
$ update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

rlenglet · 2020-04-27T05:41:43Z

Actually, nftables properly supports the REDIRECT target and the --to-ports option, cf. http://git.netfilter.org/iptables/tree/extensions/libipt_REDIRECT.c#n26. 🤔

rlenglet · 2020-04-27T05:45:40Z

nftables can actually parse and translate the rule setup by istio:

$ iptables-translate -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
nft add rule ip nat ISTIO_REDIRECT ip protocol 6 counter redirect to :15001

rlenglet · 2020-04-27T05:56:40Z

Minimal reproducer:

$ cat > /test.iptables <<EOF
* nat
-N TEST
-A TEST -p tcp -j REDIRECT --to-ports 15001
COMMIT
EOF

$ /usr/sbin/iptables-nft-restore --noflush --test --verbose /test.iptables
iptables-restore v1.8.2 (nf_tables): unknown option "--to-ports"
Error occurred at line: 3
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

It seems like iptables-nft-restore just doesn't enable nftables's REDIRECT extension:

$ cat > /test.iptables <<EOF
* nat
-N TEST
-A TEST -p tcp -j REDIRECT
COMMIT
EOF

$ /usr/sbin/iptables-nft-restore --noflush --test --verbose /test.iptables
iptables-restore v1.8.2 (nf_tables): Chain 'REDIRECT' does not exist
Error occurred at line: 3
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

This bug seems very specific to iptables-nft-restore, not nftables in general.

rlenglet · 2020-04-27T06:17:09Z

I would recommend filing this as a bug in Debian against the iptables package, which is the package that installs /usr/sbin/iptables-nft-restore, with the minimal reproducer.

rlenglet · 2020-04-27T06:37:24Z

The iptables-nft-restore command is linked to the same nftables extensions as the iptables-translate command, since they are both symlinks to the same binary:

ls -l /usr/sbin/iptables-nft-restore /usr/sbin/iptables-translate 
lrwxrwxrwx 1 root root 17 Mar  1  2019 /usr/sbin/iptables-nft-restore -> xtables-nft-multi
lrwxrwxrwx 1 root root 17 Mar  1  2019 /usr/sbin/iptables-translate -> xtables-nft-multi

So the problem would seem to be that iptables-nft-restore does not enable nftables extensions at all. That's an issue in the build upstream.

It's hard to know whether this is expected behavior, since iptables-restore is not covered by any of the upstream tests.

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279

gnz00 · 2020-04-27T17:56:20Z

I did actually try update-alternatives --set ..., but I ran into some issues which I assume are due to the lack of kernel module loading on Docker for OSX. I assume it works on a VM, but I was just looking for a way to run a standalone Istio container for a proof-of-concept so I gave up.

iptables-restore v1.8.2 (legacy): iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables-save
panic: exit status 2

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739ca, 0x10, 0xc00000cc00, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000105d30, 0x443001, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:473 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000105d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:480 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000105d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:427 +0x24e2
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xd819c0, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e030, 0x0, 0x0, 0xd5c740, 0xc00001e030)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00007a058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

rlenglet · 2020-04-27T18:58:53Z

I did actually try update-alternatives --set ..., but I ran into some issues which I assume are due to the lack of kernel module loading on Docker for OSX.

I think so. You may need to run your container as privileged for this to work? There are definitely ways to make this work, since KIND works on Docker Desktop.

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279 Co-authored-by: Romain Lenglet <rlenglet@google.com>

rlenglet · 2020-04-28T18:01:05Z

BTW, another workaround is to enable Istio CNI. It still runs individual iptables/ip6tables commands instead of iptables-restore/ip6tables-restore which seems broken in Debian Buster's iptables package.

rlenglet · 2020-04-28T18:01:38Z

I will check whether upgrading to iptables ≥1.8.3 (e.g. from buster-backports) instead of buster's 1.8.2 would fix this.

rlenglet · 2020-04-28T18:10:51Z

iptables 1.8.3-2~bpo10+1 can be installed from buster-backports:

$ echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/buster-backports.list
$ apt update
$ apt -t buster-backports install iptables

But that version too has a broken iptables-restore which doesn't enable extensions:

$ cat > /test.iptables <<EOF
* nat
-N TEST
-A TEST -p tcp -j REDIRECT
COMMIT
EOF

$ /usr/sbin/iptables-nft-restore --noflush --test --verbose /test.iptables
iptables-restore v1.8.3 (nf_tables): Chain 'REDIRECT' does not exist
Error occurred at line: 3
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

rlenglet · 2020-04-28T18:22:02Z

iptables 1.8.4-3 can be installed from testing/bullseye:

$ echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.list.d/bullseye.list
$ cat >/etc/apt/preferences.d/99bullseye <<EOF
Package: *
Pin: release n=bullseye
Pin-Priority: 50
EOF
$ cat >/etc/apt/preferences.d/99bullseye-iptables <<EOF
Package: iptables
Pin: release n=bullseye
Pin-Priority: 900
EOF
$ apt update
$ apt install iptables

That version 1.8.4-3 has a working iptables-restore which does enable extensions:

$ cat > /test.iptables <<EOF
* nat
-N TEST
-A TEST -p tcp -j REDIRECT --to-ports 15001
COMMIT
EOF

$ /usr/sbin/iptables-nft-restore --noflush --test --verbose /test.iptables

(no failure here)

So this is not an Istio bug. It's a bug in iptables versions <1.8.4. It's been fixed upstream.

I will close this bug. Feel free to file a bug against Debian Buster to ask them to upgrade iptables to version ≥1.8.4.

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes istio#23279

rlenglet added a commit that referenced this issue Apr 26, 2020

Pass --to-ports to iptables instead of --to-port

b3a67ab

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279

rlenglet mentioned this issue Apr 26, 2020

Pass --to-ports to iptables instead of --to-port #23289

Merged

rlenglet self-assigned this Apr 26, 2020

rlenglet added the area/networking label Apr 26, 2020

istio-testing closed this as completed in #23289 Apr 27, 2020

rlenglet reopened this Apr 27, 2020

rlenglet removed their assignment Apr 27, 2020

rlenglet changed the title ~~istio-sidecar.deb fails to start on Debian buster~~ istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting Apr 27, 2020

rlenglet mentioned this issue Apr 27, 2020

[release-1.5] Pass --to-ports to iptables instead of --to-port #23299

Merged

rlenglet added a commit that referenced this issue Apr 27, 2020

Pass --to-ports to iptables instead of --to-port

1777198

The iptables docs only specify --to-ports, even though vanilla iptables silently accepts --to-port. nftables's iptables is more strict and rejects --to-port. Fixes #23279

rlenglet closed this as completed Apr 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting #23279

istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting #23279

gnz00 commented Apr 26, 2020 •

edited by istio-policy-bot

Loading

howardjohn commented Apr 26, 2020

rlenglet commented Apr 26, 2020 •

edited

Loading

rlenglet commented Apr 26, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 •

edited

Loading

gnz00 commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 28, 2020

rlenglet commented Apr 28, 2020 •

edited

Loading

rlenglet commented Apr 28, 2020 •

edited

Loading

rlenglet commented Apr 28, 2020 •

edited

Loading

istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting #23279

istio-sidecar.deb fails to start on Debian buster with iptables default nftables setting #23279

Comments

gnz00 commented Apr 26, 2020 • edited by istio-policy-bot Loading

howardjohn commented Apr 26, 2020

rlenglet commented Apr 26, 2020 • edited Loading

rlenglet commented Apr 26, 2020 • edited Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 • edited Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 • edited Loading

rlenglet commented Apr 27, 2020

rlenglet commented Apr 27, 2020 • edited Loading

gnz00 commented Apr 27, 2020

rlenglet commented Apr 27, 2020

rlenglet commented Apr 28, 2020

rlenglet commented Apr 28, 2020 • edited Loading

rlenglet commented Apr 28, 2020 • edited Loading

rlenglet commented Apr 28, 2020 • edited Loading

gnz00 commented Apr 26, 2020 •

edited by istio-policy-bot

Loading

rlenglet commented Apr 26, 2020 •

edited

Loading

rlenglet commented Apr 26, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020 •

edited

Loading

rlenglet commented Apr 27, 2020 •

edited

Loading

rlenglet commented Apr 28, 2020 •

edited

Loading

rlenglet commented Apr 28, 2020 •

edited

Loading

rlenglet commented Apr 28, 2020 •

edited

Loading