/qubes-issues

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Networking/iptables anomaly on fedora-minimal-24 when "exposing qube to internet", destination port gets botched for TCP (ONLY for tcp) #2707

Closed
tonsimple opened this Issue Mar 15, 2017 · 4 comments

Comments

Assignees
No one assigned
Labels
Projects
None yet
Milestone
  Release 3.2 updates
3 participants
@tonsimple @unman @andrewdavidwong
@tonsimple

tonsimple commented Mar 15, 2017 •
edited
Edited 1 time
  • @tonsimple tonsimple edited Mar 16, 2017 (most recent)

Qubes OS version (e.g., R3.2):

3.2

Affected TemplateVMs (e.g., fedora-23, if applicable):

Discovered on Fedora-24-minimal used as firewall VM

Expected behavior:

A port can be forwarded to a qube so that server or program could listen on it to connections "from internet" if the docs are followed
https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world

Actual behavior:

Only UDP connections succeed. TCP gets lost.

Desired destination port is 16797, VPN service can and does correctly open the port (tested under windows and under ubuntu), software in the cube configured to listen on 16797

Upon enabling iptables logging, the following behavior was observed.

Inside network VM (in this case VPN proxyvm, but I did try eschewing VPN and connecting the firewall-vm to "plain" netvm, and adjusting rules accordingly... results in same stuff)

sudo iptables -t nat -I PREROUTING -i tun0 -p tcp --dport 16797 -j DNAT --to-destination 10.137.5.17

#(using a non-vpn-ed connection net-vm instead of sys-vpn, and eth0, results in exactly same behavior, so it's not a VPN issue. explicitly setting destination port a-la 10.137.5.17:16797 changes nothing)

sudo iptables -t nat -I PREROUTING -i tun0 -p tcp --dport 16797 -j LOG --log-prefix "WEIRDFORWARD" --log-level 6

#(logging packets)

Mar 16 02:13:36 sys-vpn kernel: WEIRDFORWARD IN=tun0 OUT= MAC= SRC=198.199.98.246 DST=10.114.0.48 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=52818 DF PROTO=TCP SPT=49896 DPT=16797 WINDOW=14600 RES=0x00 SYN URGP=0

Mar 16 02:13:40 sys-vpn kernel: WEIRDFORWARD IN=tun0 OUT= MAC= SRC=140.211.166.20 DST=10.114.0.48 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=9764 DF PROTO=TCP SPT=35146 DPT=16797 WINDOW=29200 RES=0x00 SYN URGP=0

Inside fedora-24-minimal based firewall VM (logging enabled for all packets that have destination IP set to IP of qube with the listening service)

Mar 16 02:13:39 smolfirewall kernel: WEIRDFORWARD IN=eth0 OUT=vif21.0 MAC=00:dd:5d:5d:6c:10:fe:ff:ff:ff:ff:ff:08:00 SRC=140.211.166.20 DST=10.137.5.17 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=48060 DF PROTO=TCP SPT=80 DPT=44886 WINDOW=58 RES=0x00 ACK FIN URGP=0

Mar 16 02:13:40 smolfirewall kernel: WEIRDFORWARD IN=eth0 OUT=vif21.0 MAC=00:dd:5d:5d:6c:10:fe:ff:ff:ff:ff:ff:08:00 SRC=140.211.166.20 DST=10.137.5.17 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=48061 DF PROTO=TCP SPT=80 DPT=44886 WINDOW=58 RES=0x00 ACK URGP=0

As you can see, destination port changed bizarrely

I can't find any rules that would change destination port (and I did check mangle table) and trying to do same with a "naked" connection through netvm also does not succeed.

Steps to reproduce the behavior:

  1. have a VPN proxyvm OR a normal net-vm (it only affects interface name anyhow for purpose of this problem)
  2. have fedora-24-minimal firewallvm connected to it (not tested other possible firewall vms)
  3. try to follow https://www.qubes-os.org/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world to expose a service sitting in qube connected to firewallvm

General notes:

The strangest part is that UDP is getting through fine.

Related issues:
@unman

This comment has been minimized.

Show comment
Hide comment
@unman

unman Mar 16, 2017

Member

At first sight, those second logs are ACK and ACK/FIN to high end ports from a webserver at 140.211.166.20, presumably in response to a SYN from that qube to the webserver.
I'm not clear why you would think that anything has been changed, bizarrely or not. Or have I completely misunderstood?
What were the OUTBOUND SYNs to which these were a response?
Member

unman commented Mar 16, 2017

At first sight, those second logs are ACK and ACK/FIN to high end ports from a webserver at 140.211.166.20, presumably in response to a SYN from that qube to the webserver.
I'm not clear why you would think that anything has been changed, bizarrely or not. Or have I completely misunderstood?
What were the OUTBOUND SYNs to which these were a response?

@andrewdavidwong andrewdavidwong added bug C: Fedora labels Mar 16, 2017

@andrewdavidwong andrewdavidwong added this to the Release 3.2 updates milestone Mar 16, 2017

@tonsimple

This comment has been minimized.

Show comment
Hide comment
@tonsimple

tonsimple Mar 16, 2017

@unman
I don't seem to be able to log any outbound traffic from the qube passing through smolfirewall, and, more peculiarly, I can't log the incoming traffic going through smolfirewall to port 16797 of the qube.

The rules I'm using on smolfirewall is

sudo iptables -I FORWARD -d 10.137.5.17 -p tcp -j LOG --log-prefix "WEIRDFORWARD" --log-level 6

sudo iptables -I FORWARD -s 10.137.5.17 -p tcp -j LOG --log-prefix "WEIRDFORWARD" --log-level 6

which should catch all traffic to 10.137.5.17 including the traffic that is supposed to go to port 16797 (not a single packet to 16797 passing through smolfirewall to 10.137.5.17 so far)

tonsimple commented Mar 16, 2017 •
edited
Edited 1 time
  • @tonsimple tonsimple edited Mar 16, 2017 (most recent)

@unman
I don't seem to be able to log any outbound traffic from the qube passing through smolfirewall, and, more peculiarly, I can't log the incoming traffic going through smolfirewall to port 16797 of the qube.

The rules I'm using on smolfirewall is

sudo iptables -I FORWARD -d 10.137.5.17 -p tcp -j LOG --log-prefix "WEIRDFORWARD" --log-level 6

sudo iptables -I FORWARD -s 10.137.5.17 -p tcp -j LOG --log-prefix "WEIRDFORWARD" --log-level 6

which should catch all traffic to 10.137.5.17 including the traffic that is supposed to go to port 16797 (not a single packet to 16797 passing through smolfirewall to 10.137.5.17 so far)
@tonsimple

This comment has been minimized.

Show comment
Hide comment
@tonsimple

tonsimple Mar 16, 2017

@unman
The issue seems to be with policy routing, but I'm too bad at understanding it to even be sure whether it's really issue with (fedora 24 minimal) smolfirewall vm .

Anyway, I seem to have found an iptables-only workaround (a packet got through and was processed by the program inside "target qube", wheeee!)
I'll test it further and report if workaround works a bit later.

tonsimple commented Mar 16, 2017 •
edited
Edited 1 time
  • @tonsimple tonsimple edited Mar 16, 2017 (most recent)

@unman
The issue seems to be with policy routing, but I'm too bad at understanding it to even be sure whether it's really issue with (fedora 24 minimal) smolfirewall vm .

Anyway, I seem to have found an iptables-only workaround (a packet got through and was processed by the program inside "target qube", wheeee!)
I'll test it further and report if workaround works a bit later.
@tonsimple

This comment has been minimized.

Show comment
Hide comment
@tonsimple

tonsimple Mar 19, 2017

Okay, issue gone after I re-installed fedora-24-minimal template "from scratch"

So it was "my bad", sorry

tonsimple commented Mar 19, 2017

Okay, issue gone after I re-installed fedora-24-minimal template "from scratch"

So it was "my bad", sorry

@tonsimple tonsimple closed this Mar 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment