Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to access Google or interact with GCE via the Google SDK (gcloud) #210

Closed
eureki opened this Issue Jan 7, 2017 · 17 comments

Comments

Projects
None yet
@eureki
Copy link

eureki commented Jan 7, 2017

OS / Environment

Ubuntu 16.04.1 LTS

Ansible version

2.2.0.0

Version of components from requirements.txt

Name: boto
Version: 2.38.0
Name: dopy
Version: 0.3.5
Name: azure
Version: 2.0.0rc5
Name: apache-libcloud
Version: 1.5.0
Name: six
Version: 1.10.0
Name: pyOpenSSL
Version: 16.2.0

Summary of the problem

I am in the process of transitioning from OpenVPN to Algo on a Google Cloud hosted instance. The Algo deployment script has been run locally on the VPN server and configured itself easily - connections can be made to Algo successfully and traffic is routable internally.

The problem is that traffic to *.google.com is not working when connected to the Algo VPN. I am also unable to use the Google SDK (gcloud) to interact with the Google Cloud Console. I am able to access other websites (github, bitbucket etc.), and services fine. A traceroute is made to google.com and Google DNS servers (8.8.8.8) without issue, however wget google.com does not work when connected to Algo. I am able to access *.google.com and use the Google SDK when connected to OpenVPN as an alternative.

I have tried amending iptables routing rules as per this and this; neither have made Google accessible. All traffic within the VPN Subnet has been opened through a Google Cloud Firewall rule.

Does Algo or the VPN instance need to be configured differently?

Steps to reproduce the behavior

  1. Connect to Algo VPN
  2. gcloud compute ssh algo-vpn (for example), alternatively
  3. Browse to google

The way of deployment (cloud or local)

Local deployment on Google Cloud instance

Expected behavior

Successful connection to the Algo VPN server via the gcloud SDK

Actual behavior

Connection to the server times out. google.com, cloud.google.com are also inaccessible.

Full log

When connected to Algo:

  • ifconfig:
    $ ifconfig
    ipsec0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1400
    inet 10.19.48.1 --> 10.19.48.1 netmask 0xff000000
    inet6 fe80::ca69:cdff:fe8e:903a%ipsec0 prefixlen 64 scopeid 0xa
    inet6 fd9d:bc11:4020::1 prefixlen 64
    nd6 options=201<PERFORMNUD,DAD>

  • traceroute
    $ traceroute google.com
    traceroute: Warning: google.com has multiple addresses; using 64.233.188.100
    traceroute to google.com (64.233.188.100), 64 hops max, 52 byte packets
    1 10.0.0.2 (10.0.0.2) 190.221 ms 190.884 ms 187.699 ms
    2 tk-in-f100.1e100.net (64.233.188.100) 188.655 ms 189.236 ms 189.405 ms

  • wget
    $ wget google.com
    --2017-01-07 17:43:55-- http://google.com/
    Resolving google.com... 64.233.188.101, 64.233.188.100, 64.233.188.138, ...
    Connecting to google.com|64.233.188.101|:80... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: http://www.google.com/ [following]
    --2017-01-07 17:43:55-- http://www.google.com/
    Resolving www.google.com... 64.233.188.147, 64.233.188.106, 64.233.188.99, ...
    Connecting to www.google.com|64.233.188.147|:80... connected.
    HTTP request sent, awaiting response...

On the VPN instance:

  • iptables -L
    root@algo-vpn:~$ iptables -L
    sudo: unable to resolve host management-vpn
    Chain INPUT (policy DROP)
    target prot opt source destination
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    ACCEPT esp -- anywhere anywhere
    ACCEPT ah -- anywhere anywhere
    DROP icmp -- anywhere anywhere icmp echo-request limit: up to 5/sec burst 5 mode srcip
    ACCEPT udp -- anywhere anywhere multiport dports isakmp,ipsec-nat-t
    ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
    ACCEPT ipencap-- anywhere anywhere policy match dir in pol ipsec proto esp
    ACCEPT udp -- anywhere 172.16.0.1 udp dpt:domain
    ACCEPT tcp -- anywhere 172.16.0.1 multiport dports http-alt,8118
    Chain FORWARD (policy DROP)
    target prot opt source destination
    DROP all -- 10.19.48.0/24 10.19.48.0/24
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DROP tcp -- anywhere anywhere tcp dpt:microsoft-ds
    DROP udp -- anywhere anywhere multiport ports netbios-ns,netbios-dgm
    DROP tcp -- anywhere anywhere multiport ports netbios-ns,netbios-ssn
    ACCEPT all -- 10.19.48.0/24 anywhere ctstate NEW policy match dir in pol ipsec
    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

  • iptables -t nat -L
    root@algo-vpn:~$ iptables -t nat -L
    Chain PREROUTING (policy ACCEPT)
    target prot opt source destination
    Chain INPUT (policy ACCEPT)
    target prot opt source destination
    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination
    Chain POSTROUTING (policy ACCEPT)
    target prot opt source destination
    MASQUERADE all -- 10.19.48.0/24 anywhere policy match dir out pol none

@dguido dguido added the bug label Jan 7, 2017

@dguido dguido added this to the 1.0 milestone Jan 7, 2017

@jackivanov

This comment has been minimized.

Copy link
Collaborator

jackivanov commented Jan 7, 2017

I cannot confirm, works well. Maybe related to #185. Try to tweak MTU/MSS

@eureki

This comment has been minimized.

Copy link
Author

eureki commented Jan 9, 2017

Thank you for your response and assistance.

I've tried tweaking the MTU/MSS as per this article and this one; neither have resolved the issue. Are you able to link to any resources that might solve the problem?

Also, to clarify, I am able to access Google ok from the Algo server, just not from my client (Mac). i.e wget is successful from the server; from my Mac it isn't.

@jackivanov

This comment has been minimized.

Copy link
Collaborator

jackivanov commented Jan 9, 2017

Try to use ping with the don't fragment flag and determine the proper MTU size

@eureki

This comment has been minimized.

Copy link
Author

eureki commented Jan 10, 2017

First, when not connected to the VPN:

$ ping -D -g 1250 -G 1256 google.com
PING google.com (216.58.199.46): (1250 ... 1256) data bytes
72 bytes from 216.58.199.46: icmp_seq=0 ttl=55 time=71.832 ms
wrong total length 92 instead of 1278
72 bytes from 216.58.199.46: icmp_seq=1 ttl=55 time=76.165 ms
wrong total length 92 instead of 1279
72 bytes from 216.58.199.46: icmp_seq=2 ttl=55 time=74.772 ms
wrong total length 92 instead of 1280
ping: sendto: Message too long
ping: sendto: Message too long
Request timeout for icmp_seq 3
ping: sendto: Message too long
Request timeout for icmp_seq 4
ping: sendto: Message too long
Request timeout for icmp_seq 5

From this I assume the MTU should be 1280. To compare, when connected to the VPN:

$ ping -D -g 1370 -G 1376 google.com
PING google.com (74.125.204.100): (1370 ... 1376) data bytes
72 bytes from 74.125.204.100: icmp_seq=0 ttl=52 time=330.455 ms
wrong total length 92 instead of 1398
72 bytes from 74.125.204.100: icmp_seq=1 ttl=52 time=212.901 ms
wrong total length 92 instead of 1399
72 bytes from 74.125.204.100: icmp_seq=2 ttl=52 time=213.108 ms
wrong total length 92 instead of 1400
ping: sendto: Message too long
ping: sendto: Message too long
Request timeout for icmp_seq 3
ping: sendto: Message too long
Request timeout for icmp_seq 4
ping: sendto: Message too long
Request timeout for icmp_seq 5

From this I assume the MTU should be set to 1400 for the VPN connection.
The following command was executed on the Algo server:

iptables -t mangle -A FORWARD -o eth0 \
-p tcp -m tcp --tcp-flags SYN,RST SYN \
-m tcpmss --mss 1361:1536 \
-j TCPMSS --set-mss 1400

The MTU was then manually set onmy Mac to 1400 with networksetup -setMTU en0 1400.

@migueldemoura

This comment has been minimized.

Copy link

migueldemoura commented Jan 28, 2017

Had the same issue on Ubuntu 16.10. Fixed it with:

iptables -t mangle -A FORWARD -o eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss 1360

Edit: Having the same problem again.

@mutemule

This comment has been minimized.

Copy link
Contributor

mutemule commented Jan 30, 2017

@migueldemoura: Can you try dropping the MSS even lower, to see if that works? Try 1200 to start.

You'll probably want to delete existing mangle rules before testing (or modify config.cfg to create a new instance with an appropriate value for max_mss.)

@jackivanov

This comment has been minimized.

Copy link
Collaborator

jackivanov commented Feb 12, 2017

MTU problems will be described in 1.1 #216

@dguido dguido modified the milestones: 1.1, 1.0 Feb 12, 2017

@jhunken

This comment has been minimized.

Copy link

jhunken commented Mar 27, 2017

I'm having the same issue. First time Algo user. Deployed Algo to Google Cloud. Able to connect to the Algo VPN and browse all sites except *.google.com sites from my Mac client.

Installed with 27e0fd0

@jhunken

This comment has been minimized.

Copy link

jhunken commented Mar 30, 2017

I just Deployed to AWS EC2 and everything works fine. Seems isolated to Google Cloud deploys.

@andrewhowdencom

This comment has been minimized.

Copy link

andrewhowdencom commented Mar 30, 2017

I am experiencing a similar issue, though not quite the same as #210 -- more #310 from @joshwardell. Can connect to the GCE instance, but no sites will respond. Interestingly, tcpdumping the interface on the VPN shows the traffic, but it appears that the traffic doesn't get routed back to my machine. (I think that's correct, I'm somewhat a networking newb)

This was verified by doing a DNS query for a domain that doesn't exist (something like foo.bar.baz), and seeing that head out across the interface -- but not receiving the response on my machine.

To further add to the weirdness, I can establish an SSH connection to the machine while not on the VPN and continue to operate it after I connect to the VPN, but cannot establish a "fresh" connection once connected.

Happy to continue debugging this any further. I am still looking at this as I have time, but figured I'd add my notes here.

@dguido

This comment has been minimized.

Copy link
Member

dguido commented Mar 30, 2017

It would help a lot if you all could start writing up the troubleshooting solutions that work on #216. That will eventually result in documentation we can publish.

@mrebersv

This comment has been minimized.

Copy link

mrebersv commented Apr 4, 2017

Was linked here through a series of closed tickets, though I question that 210 is actually the same as 310/345/whatever else.

Just to add as another data point, I'm experiencing the same GCE issues as others with my MTU set artificially low. Nother ever above 1400, but tried several steps down to 1280. Cannot get ICMP responses nor DNS responses. Haven't tried anything else since DNS won't work.

@devries

This comment has been minimized.

Copy link
Contributor

devries commented May 30, 2017

I have run into this issue connecting to certain sites, specifically google sites, through Algo VPN when I run it on GCE. I am using a Mac with OS 10.11.6 as the client, and did an install on an existing google compute engine instance running ubuntu 16.04 LTS. I tried a couple of things. First, with ICMP connections open to the server, when activating the VPN using the mobileconfig file I find my ipsec0 network device has an MTU of 1400. Running some ping sweeping I found that the maximum MTU accepted for a ping through ipsec0 to google.com is 1372.

I set the ipsec0 mtu to 1372 and everything works fine. I then tried blocking ICMP to the algo server, to see how it would default without any ICMP information. I found that ipsec0 set the MTU to 1500 in that case, which of course is worse. I fiddled with trying:

sudo iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS  --clamp-mss-to-pmtu

but that did not fix it. I tried a setting up Algo on a new digitalocean server, using the remote ansible commands and everything works fine through the digitalocean server, including google sites.

tl;dr - My report generally confirms the issue others are seeing with GCE specific installation.

@migueldemoura

This comment has been minimized.

Copy link

migueldemoura commented May 31, 2017

@mutemule, still no luck. Moved to DO in the meanwhile and have had no issues so far.

@elvizlai

This comment has been minimized.

Copy link

elvizlai commented Jun 6, 2017

set client ipsec0 mtu to 1360 solved this problem.

ifconfig ipsec0 mtu 1360

@0x1EE7

This comment has been minimized.

Copy link

0x1EE7 commented Jul 29, 2017

Running strongswan in docker container on GCE and @elvizlai s suggestion worked for me. Now how can I do the same for iOS. Is it possible to set MTU on tunnel setup by the server?

@mptorz

This comment has been minimized.

Copy link

mptorz commented Nov 18, 2018

I solved the problem on all my computers, but I cannot change the MTU setting on mobile devices (like iOS). Is there any workaround or should I just switch to amazon or digitial ocean?

Btw. I deployed with a default #max_mss setting which was 1316. I don't know if tweaking that would make any difference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.