Port Forward using iptables broken

[Joeviocoe]

Qubes OS version:

R4.0 (RC4)

Affected TemplateVMs:

fedora-25, fedora-26

Steps to reproduce the behavior:

Install fedora-26 or fedora-25 template, and have updated to latest
Configure sys-net and sys-firewall to use this template

Sys-Net:

iptables -t nat -A PREROUTING -i ens5 -p tcp -d <outside_ip_of_sys-net> --dport 2200 -j DNAT --to-destination <ip_of_sys-firewall> -m comment --comment 'PortFwd'
iptables -I FORWARD 2 -i ens5 -p tcp -d <ip_of_sys-firewall> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Sys-Firewall:

iptables -t nat -A PREROUTING -i eth0 -p tcp -d <ip_of_sys-firewall> --dport 2200 -j DNAT --to-destination <ip_of_AppVM> -m comment --comment 'PortFwd'
iptables -I FORWARD 2 -i eth0 -p tcp -d <ip_of_AppVM> --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Target AppVM:

iptables -I INPUT 5 -p tcp --dport 2200 -m conntrack --ctstate NEW -j ACCEPT -m comment --comment 'PortFwd'

Expected behavior:

sys-net & sys-firewall:

sudo iptables -vnL -t nat | grep 2200 ; sudo iptables -vnL FORWARD | grep 2200

AppVM:

sudo ifconfig eth0 | grep cast ; sudo iptables -vnL INPUT | grep 2200

SYS-NET

3   180 DNAT       tcp  --  eth0   *       0.0.0.0/0            10.0.1.53            tcp dpt:2200 /* PortFwd */ to:10.137.0.6
3   180 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            10.137.0.6           tcp dpt:2200 ctstate NEW /* PortFwd */

SYS-FIREWALL

3   180 DNAT       tcp  --  eth0   *       0.0.0.0/0            10.137.0.6           tcp dpt:2200 /* PortFwd */ to:10.137.0.17
3   180 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            10.137.0.17          tcp dpt:2200 ctstate NEW /* PortFwd */

Target AppVM

    inet 10.137.0.17  netmask 255.255.255.255  broadcast 10.255.255.255
4   220 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22000 ctstate NEW /* PortFwd */

Actual behavior:

SYS-NET

3   180 DNAT       tcp  --  eth0   *       0.0.0.0/0            10.0.1.53            tcp dpt:2200 /* PortFwd */ to:10.137.0.6
0   0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            10.137.0.6           tcp dpt:2200 ctstate NEW /* PortFwd */

SYS-FIREWALL

0   0 DNAT       tcp  --  eth0   *       0.0.0.0/0            10.137.0.6           tcp dpt:2200 /* PortFwd */ to:10.137.0.17
0   0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            10.137.0.17          tcp dpt:2200 ctstate NEW /* PortFwd */

Target AppVM

    inet 10.137.0.17  netmask 255.255.255.255  broadcast 10.255.255.255
0   0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22000 ctstate NEW /* PortFwd */

General notes:

Fedora templates have a weird issue where the packet counter on the sys-net nat FORWARD chain does not increment. The PREROUTING chain does increment.

The commands work, my configuration is correct, as it was working on R3.2 and does work on R4.0 if using an older debian-8 template.

---

Related issues:

Using Debian-8 template that came with Qubes 4.0 RC2 or earlier.... does work as expected.
It has iptables 1.4, while fedora has iptables 1.6.
I can go back and forth between fedora-25/26 and debian-8, and it will work when on debian.

Debian-9 has a weird issue where the counter on the sys-net FORWARD chain does get incremented, but nothing is sent to sys-firewall. Verified with tcpdump.

Also, when using fedora-26 as the sys-net template... it seems to block the ability to perform updates. I get "Error: failed to synchronize cache for repo" whenever I try

[Joeviocoe]

An idea: Debian don't have nftables installed by default, so
qubes-firewal fallback to iptables. But not on Fedora - there nftables
is used. This applies to both sys-net and sys-firewall.

A quick test:

1. List rules:

   nft list table ip qubes-firewall

2. Add rule accepting traffic from eth0:

   nft add rule ip qubes-firewall forward meta iifname eth0 accept

---

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

That did it!
Thanks so much for the quick resolve.

This was my results from nft list table ip qubes-firewall

table ip qubes-firewall {
	chain forward {
		type filter hook forward priority 0; policy drop;
		ct state established,related accept
		ip saddr 10.137.0.6 jump qbs-10-137-0-6
	}

	chain qbs-10-137-0-6 {
		accept
		drop
	}
}

nft add rule ip qubes-firewall forward meta iifname eth0 accept
adds iifname eth0 accept to the bottom of chain forward

Is it intended that fedora uses both iptables and nft?
Are there any security implications for allowing iifname eth0 accept (in my case for fedora-26, ens5)?

Thanks again

[Joeviocoe]

An idea: Debian don't have nftables installed by default, so
qubes-firewal fallback to iptables. But not on Fedora - there nftables
is used. This applies to both sys-net and sys-firewall.

A quick test:

1. List rules:

   nft list table ip qubes-firewall

2. Add rule accepting traffic from eth0:

   nft add rule ip qubes-firewall forward meta iifname eth0 accept

---

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

That did it!
Thanks so much for the quick resolve.

This was my results from nft list table ip qubes-firewall

table ip qubes-firewall {
	chain forward {
		type filter hook forward priority 0; policy drop;
		ct state established,related accept
		ip saddr 10.137.0.6 jump qbs-10-137-0-6
	}

	chain qbs-10-137-0-6 {
		accept
		drop
	}
}

nft add rule ip qubes-firewall forward meta iifname eth0 accept
adds iifname eth0 accept to the bottom of chain forward

Is it intended that fedora uses both iptables and nft?
Are there any security implications for allowing iifname eth0 accept (in my case for fedora-26, ens5)?

Thanks again

[Joeviocoe]

Created an updated version of a Port Forwarding script for Qubes 4.0 (RC4 tested)(https://gist.github.com/Joeviocoe/6c4dc0c283f6d6c5b1a3f5af8793292b)
qvm-portfwd <vm> <port> <proto> | <vm> clear all
Example: qvm-portfwd webserv 8888 tcp

Command line specify the "VM, Port and Protocol"... or just "VM clear all" to undo previous.
Script will recursively configure iptables/nft for all proxyVMs in use.
Now uses comments on iptables to remove previous entries (no duplicates)

Works with Fedora 25/26 which uses nft rules along with iptables
Works with Debian 8/9 too

[Joeviocoe]

Created an updated version of a Port Forwarding script for Qubes 4.0 (RC4 tested)(https://gist.github.com/Joeviocoe/6c4dc0c283f6d6c5b1a3f5af8793292b)
qvm-portfwd <vm> <port> <proto> | <vm> clear all
Example: qvm-portfwd webserv 8888 tcp

Command line specify the "VM, Port and Protocol"... or just "VM clear all" to undo previous.
Script will recursively configure iptables/nft for all proxyVMs in use.
Now uses comments on iptables to remove previous entries (no duplicates)

Works with Fedora 25/26 which uses nft rules along with iptables
Works with Debian 8/9 too

[marmarek]

BTW for TCP it's possible to use qrexec instead of all those firewall rules: see #2148

[marmarek]

BTW for TCP it's possible to use qrexec instead of all those firewall rules: see #2148

[Joeviocoe]

Thanks... socat is awesome. Edit: The qrexec method is cleaner, and better since it doesn't need to recurse through proxyVMs like sys-firewall. Also it doesn't require the target VM to have any netvm assigned (which is good for secure VMs. In addition to the instructions on #2148, a single iptables rule is needed for the Source VM (sys-net) INPUT chain to ACCEPT the initial connection.

…

On Fri, Feb 9, 2018 at 11:10 PM, Marek Marczykowski-Górecki < ***@***.***> wrote: BTW for TCP it's possible to use qrexec instead of all those firewall rules: see #2148 <#2148> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3556 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APt0QT7CDTisZ4CTZ0ymI6f8PAWM5hXIks5tTRa-gaJpZM4R_d3s> .

[Joeviocoe]

Thanks... socat is awesome. Edit: The qrexec method is cleaner, and better since it doesn't need to recurse through proxyVMs like sys-firewall. Also it doesn't require the target VM to have any netvm assigned (which is good for secure VMs. In addition to the instructions on #2148, a single iptables rule is needed for the Source VM (sys-net) INPUT chain to ACCEPT the initial connection.

…

On Fri, Feb 9, 2018 at 11:10 PM, Marek Marczykowski-Górecki < ***@***.***> wrote: BTW for TCP it's possible to use qrexec instead of all those firewall rules: see #2148 <#2148> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3556 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APt0QT7CDTisZ4CTZ0ymI6f8PAWM5hXIks5tTRa-gaJpZM4R_d3s> .

[andrewdavidwong]

Closing this as "resolved." If you believe the issue is not yet resolved, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen this. Thank you.

[andrewdavidwong]

Closing this as "resolved." If you believe the issue is not yet resolved, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen this. Thank you.

[Joeviocoe]

The qrexec/socat method is cleaner, and better since it doesn't need to recurse through proxyVMs like sys-firewall. Also it doesn't require the target VM to have any netvm assigned (which is good for secure VMs).

Using socat (great for tcp only connections)
https://gist.github.com/Joeviocoe/90ec9fd9a0769b4671a8ae9c87584187

Thanks again.

[Joeviocoe]

The qrexec/socat method is cleaner, and better since it doesn't need to recurse through proxyVMs like sys-firewall. Also it doesn't require the target VM to have any netvm assigned (which is good for secure VMs).

Using socat (great for tcp only connections)
https://gist.github.com/Joeviocoe/90ec9fd9a0769b4671a8ae9c87584187

Thanks again.

[awokd]

@Joeviocoe Any chance you could summarize the above and update https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world for R4.0?

[awokd]

@Joeviocoe Any chance you could summarize the above and update https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world for R4.0?

[tlaurion]

@Joeviocoe @marmarek : It would be awesome if a generic solution, like Qubes Network Server, would be introduced into Qubes.

It was nicely taking care of inter vm connectivity ("from-" prepending rules) also permitting to expose a vm to the lan (and the internet if desired) directly from the firewall GUI, with the exception of needing to call static_ip script in dom0 to expose it to the lan. I miss that!

[tlaurion]

@Joeviocoe @marmarek : It would be awesome if a generic solution, like Qubes Network Server, would be introduced into Qubes.

It was nicely taking care of inter vm connectivity ("from-" prepending rules) also permitting to expose a vm to the lan (and the internet if desired) directly from the firewall GUI, with the exception of needing to call static_ip script in dom0 to expose it to the lan. I miss that!

[adubois]

PR-605 to address the documentation created.

[adubois]

PR-605 to address the documentation created.

[yonjah]

Tried to follow all the GISTs but couldn't really get inter-VM communication going.
I need to have multiple machines connected using NFS so qrexec is not an option.
What are the iptables/nfttables rules needed to get two VMs talking to each other ?

[yonjah]

Tried to follow all the GISTs but couldn't really get inter-VM communication going.
I need to have multiple machines connected using NFS so qrexec is not an option.
What are the iptables/nfttables rules needed to get two VMs talking to each other ?

[awokd]

@yonjah Please see adubois's updated documentation in https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes. Often the qubes-users mailing list is a better place to ask these types of questions unless you don't mind a potentially very long wait!

[awokd]

@yonjah Please see adubois's updated documentation in https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes. Often the qubes-users mailing list is a better place to ask these types of questions unless you don't mind a potentially very long wait!

[yonjah]

@awokd Sorry the docs reference a github issue that eventually led me down to the doc change and this issue so I thought this is the best place to ask this issue since the docs didn't work for me (and I think they are identical to R3.x).
After doing a bit of more digging I found out that the difference is that my VMs are blocked from accessing the outside world. for some reason in R4.0 this breaks the interVM communication as well.
If i explicitly add the serverVM ip as whitelisted I can only ping it but I'm unable to do actual tcp requests
If I allow the clientVM to have full network access than I can also get tcp connections going.
I really want to prevent VMs from having external network access so I hope there is a way to only allow internal access

[yonjah]

@awokd Sorry the docs reference a github issue that eventually led me down to the doc change and this issue so I thought this is the best place to ask this issue since the docs didn't work for me (and I think they are identical to R3.x).
After doing a bit of more digging I found out that the difference is that my VMs are blocked from accessing the outside world. for some reason in R4.0 this breaks the interVM communication as well.
If i explicitly add the serverVM ip as whitelisted I can only ping it but I'm unable to do actual tcp requests
If I allow the clientVM to have full network access than I can also get tcp connections going.
I really want to prevent VMs from having external network access so I hope there is a way to only allow internal access

[awokd]

@yonjah No problem, just hoping to save you some time! I don't think anyone monitors comments on closed issues unless they forgot to unsubscribe from them like I did this one. :) If you think this issue is not resolved and should be re-opened, you can ping @ andrewdavidwong and ask.

[awokd]

@yonjah No problem, just hoping to save you some time! I don't think anyone monitors comments on closed issues unless they forgot to unsubscribe from them like I did this one. :) If you think this issue is not resolved and should be re-opened, you can ping @ andrewdavidwong and ask.

[andrewdavidwong]

I don't think anyone monitors comments on closed issues unless they forgot to unsubscribe from them like I did this one. :)

Don't worry. I monitor all comments on all issues. :)

If you think this issue is not resolved and should be re-opened, you can ping @ andrewdavidwong and ask.

I didn't reopen because it sounded like @yonjah's question was asking for advice rather than reporting that the original problem still exists or that there was a regression.

@yonjah, if you think that this issue is not truly resolved, please say so, and provide some steps to reproduce, if possible.

[andrewdavidwong]

I don't think anyone monitors comments on closed issues unless they forgot to unsubscribe from them like I did this one. :)

Don't worry. I monitor all comments on all issues. :)

If you think this issue is not resolved and should be re-opened, you can ping @ andrewdavidwong and ask.

I didn't reopen because it sounded like @yonjah's question was asking for advice rather than reporting that the original problem still exists or that there was a regression.

@yonjah, if you think that this issue is not truly resolved, please say so, and provide some steps to reproduce, if possible.

[yonjah]

@awokd No worries. Responses here were much faster than I had time to address any way :-)

@andrewdavidwong I think this works fine for me now.
It might be worth mentioning that if you block outside connections the VM you need to whitelist the target VM (since this wasn't needed on R3).
Tho after white listing networking works fine for some reason qubes-manager some time didn't actually change the rules. Not really sure how to reproduce it but I'll try to see if it happens again.

[yonjah]

@awokd No worries. Responses here were much faster than I had time to address any way :-)

@andrewdavidwong I think this works fine for me now.
It might be worth mentioning that if you block outside connections the VM you need to whitelist the target VM (since this wasn't needed on R3).
Tho after white listing networking works fine for some reason qubes-manager some time didn't actually change the rules. Not really sure how to reproduce it but I'll try to see if it happens again.