New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use Minikube on VPN #1099

Closed
blockloop opened this Issue Feb 7, 2017 · 23 comments

Comments

Projects
None yet
10 participants
@blockloop
Copy link

blockloop commented Feb 7, 2017

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Minikube version (use minikube version): 0.16

Environment:

  • OS (e.g. from /etc/os-release): macOS
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox AND vmwarefusion
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep ISO): file:///Users/bjones/.minikube/cache/iso/minikube-v1.0.6.iso
  • Install tools:
  • Others:

What happened:
Creating and using minikube works fine. Open Cisco AnyConnect Secure Mobility Client (VPN client for work) and minikube no longer works. minikube start hangs, kubectl commands cannot reach the host.

What you expected to happen:
Minikube should work while connected to a VPN

How to reproduce it (as minimally and precisely as possible):

  1. Create minikube off of VPN
  2. Connect to VPN
  3. Try to use minikube

Anything else do we need to know:

I'm also not able to create a minikube with minikube start while I'm on the VPN. When I try to create a minikube instance while connected to a VPN network I get the following error

E0207 10:00:19.260681    9862 start.go:96] Error starting host: Error creating host: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: i/o timeout

I've tried using the --host-only-cidr 10.10.10.1/24 to avoid conflicts in IP ranges, but that didn't help.

Here is the output of minikube start -v 7 when I have the VPN enabled and trying to connect to an existing minikube instance.

$ minikube start -v 7
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/minikube
Launching plugin server for driver vmwarefusion
Plugin server listening at address 127.0.0.1:57463
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) DBG | executing: /Applications/VMware Fusion.app/Contents/Library/vmrun list
(minikube) Calling .Start
(minikube) DBG | executing: /Applications/VMware Fusion.app/Contents/Library/vmrun start /Users/bjones/.minikube/machines/minikube/minikube.vmx nogui
(minikube) DBG | Mounting Shared Folders...
(minikube) DBG | executing: /Applications/VMware Fusion.app/Contents/Library/vmrun -gu docker -gp tcuser runScriptInGuest /Users/bjones/.minikube/machines/minikube/minikube.vmx /bin/sh [ ! -d /Users ]&& sudo mkdir /Users; sudo mount --bind /mnt/hgfs//Users /Users || [ -f /usr/local/bin/vmhgfs-fuse ]&& sudo /usr/local/bin/vmhgfs-fuse -o allow_other .host:/Users /Users || sudo mount -t vmhgfs -o uid=$(id -u),gid=$(id -g) .host:/Users /Users
(minikube) Calling .GetConfigRaw
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG | executing: /Applications/VMware Fusion.app/Contents/Library/vmrun list
(minikube) DBG | MAC address in VMX: 00:0c:29:5b:38:e2
(minikube) DBG | Trying to find IP address in configuration file: /Library/Preferences/VMware Fusion/vmnet1/dhcpd.conf
(minikube) DBG | Following IPs found map[00:50:56:c0:00:01:172.16.30.1]
(minikube) DBG | Trying to find IP address in configuration file: /Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf
(minikube) DBG | Following IPs found map[00:50:56:c0:00:08:172.16.9.1]
(minikube) DBG | Trying to find IP address in leases file: /var/db/vmware/vmnet-dhcpd-vmnet1.leases
(minikube) DBG | Trying to find IP address in leases file: /var/db/vmware/vmnet-dhcpd-vmnet8.leases
(minikube) DBG | IP found in DHCP lease table: 172.16.9.131
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/bjones/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.9.131 -o IdentitiesOnly=yes -i /Users/bjones/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
command : exit 0
err     : exit status 255
output  :

Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG | executing: /Applications/VMware Fusion.app/Contents/Library/vmrun list
(minikube) DBG | MAC address in VMX: 00:0c:29:5b:38:e2
(minikube) DBG | Trying to find IP address in configuration file: /Library/Preferences/VMware Fusion/vmnet1/dhcpd.conf
(minikube) DBG | Following IPs found map[00:50:56:c0:00:01:172.16.30.1]
(minikube) DBG | Trying to find IP address in configuration file: /Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf
(minikube) DBG | Following IPs found map[00:50:56:c0:00:08:172.16.9.1]
(minikube) DBG | Trying to find IP address in leases file: /var/db/vmware/vmnet-dhcpd-vmnet1.leases
(minikube) DBG | Trying to find IP address in leases file: /var/db/vmware/vmnet-dhcpd-vmnet8.leases
(minikube) DBG | IP found in DHCP lease table: 172.16.9.131
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/bjones/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@172.16.9.131 -o IdentitiesOnly=yes -i /Users/bjones/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
^C

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 7, 2017

When I switched to the xhyve driver I can start the instance just fine, but I still cannot interact using kubectl, minikube status, minikube dashboard, etc although I can minikube ssh and see the docker containers running.

@r2d4

This comment has been minimized.

Copy link
Member

r2d4 commented Feb 7, 2017

You should make sure that you are on the latest minikube version
minikube version should be 0.16

This error usually means that docker took too long to start up. I think we've made this faster in the latest version.

E0207 10:00:19.260681    9862 start.go:96] Error starting host: Error creating host: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: i/o timeout

If the latest version doesn't work, can you run a minikube logs and post anything relevant here? What error does kubetl get pods give?

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 7, 2017

I updated minikube, sorry. I forgot to update the OP version. There doesn't seem to be anything wrong with the running vm. I think the problem is related to networking from host to vm. When I disable my VPN I can interact with minikube just fine, but when the VPN is connected I get the following:

$ kubectl get pods
Unable to connect to the server: dial tcp 192.168.99.100:8443: i/o timeout

minikube dashboard just sits there waiting and never terminates.

minikube ssh works just fine on and off the VPN

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 7, 2017

Weird. I believe this is an issue with this Cisco VPN Client and has nothing to do with minikube.

I'll close the issue and look into it. Thanks for entertaining me.

@blockloop blockloop closed this Feb 7, 2017

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 7, 2017

I've looked more into this and I still think there might be an issue with the networking in minikube. When my VPN client has Tunnel Mode set to Tunnel All I cannot access anything on the minikube instance, but I can minikube ssh just fine and see everything running. Minikube set everything up properly and works just fine.

Here is the output of minikube ssh -v7 while connected to the VPN

$ minikube ssh -v7
Found binary path at /usr/local/bin/minikube
Launching plugin server for driver virtualbox
Plugin server listening at address 127.0.0.1:53432
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) DBG | COMMAND: /usr/local/bin/VBoxManage showvminfo minikube --machinereadable
(minikube) DBG | STDOUT:
(minikube) DBG | {
(minikube) DBG | name="minikube"
(minikube) DBG | groups="/"
(minikube) DBG | ostype="Linux 2.6 / 3.x / 4.x (64-bit)"
(minikube) DBG | UUID="7053e881-54b1-45f9-af66-c9a71a749a1c"
(minikube) DBG | CfgFile="/Users/bjones/.minikube/machines/minikube/minikube/minikube.vbox"
(minikube) DBG | SnapFldr="/Users/bjones/.minikube/machines/minikube/minikube/Snapshots"
(minikube) DBG | LogFldr="/Users/bjones/.minikube/machines/minikube/minikube/Logs"
(minikube) DBG | hardwareuuid="7053e881-54b1-45f9-af66-c9a71a749a1c"
(minikube) DBG | memory=3584
(minikube) DBG | pagefusion="off"
(minikube) DBG | vram=8
(minikube) DBG | cpuexecutioncap=100
(minikube) DBG | hpet="on"
(minikube) DBG | chipset="piix3"
(minikube) DBG | firmware="BIOS"
(minikube) DBG | cpus=2
(minikube) DBG | pae="on"
(minikube) DBG | longmode="on"
(minikube) DBG | cpuid-portability-level=0
(minikube) DBG | bootmenu="disabled"
(minikube) DBG | boot1="dvd"
(minikube) DBG | boot2="dvd"
(minikube) DBG | boot3="disk"
(minikube) DBG | boot4="none"
(minikube) DBG | acpi="on"
(minikube) DBG | ioapic="on"
(minikube) DBG | biossystemtimeoffset=0
(minikube) DBG | rtcuseutc="on"
(minikube) DBG | hwvirtex="on"
(minikube) DBG | nestedpaging="on"
(minikube) DBG | largepages="on"
(minikube) DBG | vtxvpid="on"
(minikube) DBG | vtxux="on"
(minikube) DBG | paravirtprovider="default"
(minikube) DBG | VMState="running"
(minikube) DBG | VMStateChangeTime="2017-02-07T18:36:44.116000000"
(minikube) DBG | monitorcount=1
(minikube) DBG | accelerate3d="off"
(minikube) DBG | accelerate2dvideo="off"
(minikube) DBG | teleporterenabled="off"
(minikube) DBG | teleporterport=0
(minikube) DBG | teleporteraddress=""
(minikube) DBG | teleporterpassword=""
(minikube) DBG | tracing-enabled="off"
(minikube) DBG | tracing-allow-vm-access="off"
(minikube) Calling .GetSSHHostname
(minikube) DBG | tracing-config=""
(minikube) DBG | autostart-enabled="off"
(minikube) DBG | autostart-delay=0
(minikube) DBG | defaultfrontend=""
(minikube) DBG | storagecontrollername0="SATA"
(minikube) DBG | storagecontrollertype0="IntelAhci"
(minikube) DBG | storagecontrollerinstance0="0"
(minikube) Calling .GetSSHPort
(minikube) DBG | storagecontrollermaxportcount0="30"
(minikube) DBG | storagecontrollerportcount0="30"
(minikube) DBG | storagecontrollerbootable0="on"
(minikube) DBG | "SATA-0-0"="/Users/bjones/.minikube/machines/minikube/boot2docker.iso"
(minikube) DBG | "SATA-ImageUUID-0-0"="4f8f2041-b9b4-4355-adb6-2fc6d4aa85dc"
(minikube) DBG | "SATA-tempeject"="off"
(minikube) DBG | "SATA-IsEjected"="off"
(minikube) DBG | "SATA-1-0"="/Users/bjones/.minikube/machines/minikube/disk.vmdk"
(minikube) DBG | "SATA-ImageUUID-1-0"="404542e8-ec01-4f2a-a8fc-1ee3b83c8b69"
(minikube) DBG | "SATA-2-0"="none"
(minikube) DBG | "SATA-3-0"="none"
(minikube) DBG | "SATA-4-0"="none"
(minikube) DBG | "SATA-5-0"="none"
(minikube) DBG | "SATA-6-0"="none"
(minikube) DBG | "SATA-7-0"="none"
(minikube) DBG | "SATA-8-0"="none"
(minikube) DBG | "SATA-9-0"="none"
(minikube) DBG | "SATA-10-0"="none"
(minikube) DBG | "SATA-11-0"="none"
(minikube) DBG | "SATA-12-0"="none"
(minikube) DBG | "SATA-13-0"="none"
(minikube) Calling .GetSSHKeyPath
(minikube) DBG | "SATA-14-0"="none"
(minikube) DBG | "SATA-15-0"="none"
(minikube) DBG | "SATA-16-0"="none"
(minikube) DBG | "SATA-17-0"="none"
(minikube) DBG | "SATA-18-0"="none"
(minikube) DBG | "SATA-19-0"="none"
(minikube) DBG | "SATA-20-0"="none"
(minikube) DBG | "SATA-21-0"="none"
(minikube) DBG | "SATA-22-0"="none"
(minikube) DBG | "SATA-23-0"="none"
(minikube) DBG | "SATA-24-0"="none"
(minikube) DBG | "SATA-25-0"="none"
(minikube) DBG | "SATA-26-0"="none"
(minikube) DBG | "SATA-27-0"="none"
(minikube) DBG | "SATA-28-0"="none"
(minikube) DBG | "SATA-29-0"="none"
(minikube) DBG | natnet1="nat"
(minikube) DBG | macaddress1="0800279E777E"
(minikube) DBG | cableconnected1="on"
(minikube) DBG | nic1="nat"
(minikube) DBG | nictype1="82540EM"
(minikube) DBG | nicspeed1="0"
(minikube) DBG | mtu="0"
(minikube) Calling .GetSSHKeyPath
(minikube) DBG | sockSnd="64"
(minikube) DBG | sockRcv="64"
(minikube) DBG | tcpWndSnd="64"
(minikube) DBG | tcpWndRcv="64"
(minikube) Calling .GetSSHUsername
(minikube) DBG | Forwarding(0)="ssh,tcp,127.0.0.1,50763,,22"
(minikube) DBG | hostonlyadapter2="vboxnet0"
(minikube) DBG | macaddress2="080027980E72"
(minikube) DBG | cableconnected2="on"
(minikube) DBG | nic2="hostonly"
(minikube) DBG | nictype2="82540EM"
(minikube) DBG | nicspeed2="0"
(minikube) DBG | nic3="none"
(minikube) DBG | nic4="none"
(minikube) DBG | nic5="none"
(minikube) DBG | nic6="none"
(minikube) DBG | nic7="none"
(minikube) DBG | nic8="none"
(minikube) DBG | hidpointing="ps2mouse"
(minikube) DBG | hidkeyboard="ps2kbd"
(minikube) DBG | uart1="off"
(minikube) DBG | uart2="off"
(minikube) DBG | uart3="off"
(minikube) DBG | uart4="off"
(minikube) DBG | lpt1="off"
(minikube) DBG | lpt2="off"
(minikube) DBG | audio="none"
(minikube) DBG | clipboard="disabled"
(minikube) DBG | draganddrop="disabled"
(minikube) DBG | SessionName="headless"
(minikube) DBG | VideoMode="720,400,0"@0,0 1
(minikube) DBG | vrde="off"
(minikube) DBG | usb="off"
Using SSH client type: external
(minikube) DBG | ehci="off"
(minikube) DBG | xhci="off"
(minikube) DBG | SharedFolderNameMachineMapping1="Users"
Using SSH private key: /Users/bjones/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | SharedFolderPathMachineMapping1="/Users"
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@127.0.0.1 -o IdentitiesOnly=yes -i /Users/bjones/.minikube/machines/minikube/id_rsa -p 50763] /usr/bin/ssh <nil>}
(minikube) DBG | VRDEActiveConnection="off"
&{/usr/bin/ssh [/usr/bin/ssh -F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@127.0.0.1 -o IdentitiesOnly=yes -i /Users/bjones/.minikube/machines/minikube/id_rsa -p 50763 ] []  <nil> <nil> <nil> [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>}
(minikube) DBG | VRDEClients=0
(minikube) DBG | vcpenabled="off"
(minikube) DBG | vcpscreens=0
(minikube) DBG | vcpfile="/Users/bjones/.minikube/machines/minikube/minikube/minikube.webm"
(minikube) DBG | vcpwidth=1024
(minikube) DBG | vcpheight=768
(minikube) DBG | vcprate=512
(minikube) DBG | vcpfps=25
(minikube) DBG | GuestMemoryBalloon=0
(minikube) DBG | GuestOSType="Linux26_64"
(minikube) DBG | GuestAdditionsRunLevel=2
(minikube) DBG | GuestAdditionsVersion="5.1.6 r110634"
(minikube) DBG | GuestAdditionsFacility_VirtualBox Base Driver=50,1486490700803
(minikube) DBG | GuestAdditionsFacility_VirtualBox System Service=50,1486490701278
(minikube) DBG | GuestAdditionsFacility_Seamless Mode=0,1486490700801
(minikube) DBG | GuestAdditionsFacility_Graphics Mode=0,1486490700801
(minikube) DBG | }
(minikube) DBG | STDERR:
(minikube) DBG | {
(minikube) DBG | }
$

@blockloop blockloop reopened this Feb 7, 2017

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 8, 2017

Looks like this is related boot2docker/boot2docker#628

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 8, 2017

After hours of searching and troubleshooting I've determined that It has something to do with Cisco AnyConnect VPN client routing traffic poorly.

@blockloop blockloop closed this Feb 8, 2017

@vhosakot

This comment has been minimized.

Copy link

vhosakot commented Feb 8, 2017

@blockloop I see the exact same issue. I cannot use minikube when I'm connected to the Cisco AnyConnect VPN 😞.

Here are my minukube configs:

$ cat ~/.minikube/machines/minikube/config.json | grep -i 'DriverName\|docker'
        "SSHUser": "docker",
        "Boot2DockerURL": "file:///Users/vhosakot/.minikube/cache/iso/minikube-v1.0.6.iso",
        "BootCmd": "loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube",
    "DriverName": "xhyve",

Here are the logs I see when I start minikube in debug mode. Looks like it just hangs indefinitely when SSH'ing.

$ minikube start --vm-driver=xhyve -v 10
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:57178
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) Calling .Start
(minikube) DBG | [xhyve -A -U DDA13938-6F9E-4519-89E2-AA1E54272733 -c 2 -m 2048M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3:0,ahci-cd,/Users/vhosakot/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk2 -f kexec,/Users/vhosakot/.minikube/machines/minikube/bzImage,/Users/vhosakot/.minikube/machines/minikube/initrd,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/vhosakot/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG | 192.168.64.2
(minikube) DBG | IP found in DHCP lease table: 192.168.64.2
(minikube) DBG | Got an ip: 192.168.64.2
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Error getting ssh command 'exit 0' : Host is not running
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys002 in order to connect to your VM
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | Using SSH private key: /Users/vhosakot/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | &{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.2 -o IdentitiesOnly=yes -i /Users/vhosakot/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | rdmsr to register 0x34 on vcpu 1
(minikube) DBG | SSH cmd err, output: exit status 255: 
(minikube) DBG | Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
(minikube) DBG | command : exit 0
(minikube) DBG | err     : exit status 255
(minikube) DBG | output  : 
(minikube) DBG | 
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | Using SSH private key: /Users/vhosakot/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | &{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.2 -o IdentitiesOnly=yes -i /Users/vhosakot/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
^C <--- Pressed CTRL+C to exit.

I see above the exit status 255 when minikube does SSH.

Interestingly, I see in the Cisco AnyConnect VPN logs that it flaps every time I start minikube.

    12:45:51 AM    Reconnecting to Boxborough - SSL...
    12:45:53 AM    Establishing VPN - Examining system...
    12:45:53 AM    Establishing VPN - Activating VPN adapter...
    12:45:53 AM    Establishing VPN - Configuring system...
    12:45:56 AM    Establishing VPN...
    12:45:56 AM    Connected to Boxborough - SSL.
    12:46:32 AM    Reconnecting to Boxborough - SSL...
    12:46:34 AM    Establishing VPN - Examining system...
    12:46:34 AM    Establishing VPN - Activating VPN adapter...
    12:46:34 AM    Establishing VPN - Configuring system...
    12:46:36 AM    Establishing VPN...
    12:46:36 AM    Connected to Boxborough - SSL.
    12:49:55 AM    Reconnecting to Boxborough - SSL...
    12:49:57 AM    Establishing VPN - Examining system...
    12:49:57 AM    Establishing VPN - Activating VPN adapter...
    12:49:57 AM    Establishing VPN - Configuring system...
    12:49:59 AM    Establishing VPN...
    12:49:59 AM    Connected to Boxborough - SSL.
    12:51:11 AM    Reconnecting to Boxborough - SSL...
    12:51:13 AM    Establishing VPN - Examining system...
    12:51:13 AM    Establishing VPN - Activating VPN adapter...
    12:51:13 AM    Establishing VPN - Configuring system...
    12:51:15 AM    Establishing VPN...
    12:51:16 AM    Connected to Boxborough - SSL.
    12:52:38 AM    Reconnecting to Boxborough - SSL...
    12:52:40 AM    Establishing VPN - Examining system...
    12:52:40 AM    Establishing VPN - Activating VPN adapter...
    12:52:40 AM    Establishing VPN - Configuring system...
    12:52:43 AM    Establishing VPN...
    12:52:43 AM    Connected to Boxborough - SSL.
    12:56:46 AM    Reconnecting to Boxborough - SSL...
    12:56:48 AM    Establishing VPN - Examining system...
    12:56:48 AM    Establishing VPN - Activating VPN adapter...
    12:56:48 AM    Establishing VPN - Configuring system...
    12:56:50 AM    Establishing VPN...
    12:56:50 AM    Connected to Boxborough - SSL.

Is this issue due to boot2docker/boot2docker#628? Is there a work-around? Should we ask Cisco's AnyConnect VPN team to fix this?

@blockloop

This comment has been minimized.

Copy link

blockloop commented Feb 8, 2017

Browsing around I've discovered that the Cisco AnyConnect App supposedly wreaks havoc on the iptables. For some reason I can ssh to the machine but I cannot interact with it in the browser or even use minikube dashboard. If you have the ability to use a split tunnel then that fixes this problem, but then I cannot access any of my VPN network from within the VM.

@vhosakot

This comment has been minimized.

Copy link

vhosakot commented Feb 8, 2017

Thanks for the info @blockloop. I'll see if I can setup a split tunnel.

@vhosakot

This comment has been minimized.

Copy link

vhosakot commented Feb 8, 2017

I generated the DART (Diagnostic And Reporting Tool) report from Cisco AnyConnect VPN, and see these errors in system.log. Clearly, we can see below docker is flapping Cisco AnyConnect VPN.

Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Adding filter rule: pass in log quick on awdl0 inet6 proto ipv6-icmp from ::0 to FF02:0:0:0:0:1:FF50:A62A icmp6-type 135 code 0
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Adding filter rule: pass out log quick on awdl0 inet6 proto ipv6-icmp from FE80:0:0:0:50D4:61FF:FE50:A62A to FF02::1 icmp6-type 136 code 0
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Adding filter rule: pass in log quick on awdl0 inet6 proto ipv6-icmp from any to FF02::1 icmp6-type 136 code 0
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Adding filter rule: block log quick inet all ! tagged cisco_anyconnect_vpn_pass
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Adding filter rule: block log quick inet6 all ! tagged cisco_anyconnect_vpn_pass
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: saveRulesToConfigFile File: ../../vpn/AgentUtilities/UnixFwUtil_OSX.cpp Line: 758 Updating pf configuration file /opt/cisco/anyconnect/ac_pf_final.conf with: 0 options, 0 scrubs, 1 IPv4 rules, 1 IPv6 rules
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: saveRulesToConfigFile File: ../../vpn/AgentUtilities/UnixFwUtil_OSX.cpp Line: 758 Updating pf configuration file /opt/cisco/anyconnect/ac_pf_filt.conf with: 1 options, 1 scrubs, 11 IPv4 rules, 43 IPv6 rules
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: enablePf File: ../../vpn/AgentUtilities/UnixFwUtil_OSX.cpp Line: 1165 Command '/sbin/pfctl -E -f /etc/pf.conf 2>&1' returned output pfctl: Use of -f option, could result in flushing of rules present in the main ruleset added by the system at startup. See /etc/pf.conf for further details.  No ALTQ support in kernel ALTQ related functions disabled pf enabled Token : 18380007534171971645  , extracted token 18380007534171971645
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: reloadPfRules File: ../../vpn/AgentUtilities/UnixFwUtil_OSX.cpp Line: 1251 Command '/sbin/pfctl -f /etc/pf.conf 2>&1' returned output pfctl: Use of -f option, could result in flushing of rules present in the main ruleset added by the system at startup. See /etc/pf.conf for further details.  No ALTQ support in kernel ALTQ related functions disabled 
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: applyFirewallConfiguration File: ../../vpn/AgentUtilities/HostConfigMgr.cpp Line: 900 No Firewall Rules to configure
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: The network control state changed to restricted.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Message type information sent to the user: Establishing VPN...
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnTunnelStateChange File: ../../vpn/Agent/TND.cpp Line: 1970 tunnel state change notification (new 1, old 0)
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: The VPN connection has been established and can now pass data.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: VPN state: Connected Network state: Network Accessible Network control state: Network Access: Restricted Network type: Untrusted
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: The Primary DTLS connection to the secure gateway is being established.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnTunnelStateChange File: ../../vpn/Agent/TND.cpp Line: 1970 tunnel state change notification (new 1, old 1)
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: connectTransport File: ../../vpn/Common/IPC/SocketTransport.cpp Line: 856 Invoked Function: ::bind Return Code: 22 (0x00000016) Description: unknown 
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: initiateTransport File: ../../vpn/Agent/DtlsTunnelTransport.cpp Line: 222 Opened DTLS socket from [192.168.0.29]:64858 to [198.135.0.166]:443
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpndownloader[8247]: Function: WaitForCompletion File: /tmp/build/thehoff/Ironman2_MR30.980020412858/Ironman2_MR3/vpn/Common/Utility/Thread.cpp Line: 286 The thread has successfully completed execution.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpndownloader[8247]: Cisco AnyConnect Secure Mobility Client Downloader (VPN) exiting, version 4.3.03086 , return code 0 [0x00000000]
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Function: launchCachedDownloader File: ../../vpn/Api/ConnectMgr.cpp Line: 7016 Invoked Function: ConnectMgr :: launchCachedDownloader Return Code: 0 (0x00000000) Description: Cached Downloader terminated normally 
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: VPN state: Connected Network state: Network Accessible Network control state: Network Access: Restricted Network type: Untrusted
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Message type information sent to the user: Connected to Boxborough - SSL.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Launching script "/opt/cisco/anyconnect/script/OnConnect_Mac_AnyConnect_Screenlock_D13v2.sh".
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: A DTLS connection has been established using cipher AES256-SHA
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: The Primary DTLS connection to the secure gateway has been established.
Feb  8 00:45:32 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnTunnelStateChange File: ../../vpn/Agent/TND.cpp Line: 1970 tunnel state change notification (new 1, old 1)
Feb  8 00:45:34 VHOSAKOT-M-H6X5 Docker[814]: hosts file has bindings for localhost broadcasthost localhost
Feb  8 00:45:34 VHOSAKOT-M-H6X5 defaults[8385]: 
	The domain/default pair of (/Library/Preferences/com.jamfsoftware.jamf, global_log_level) does not exist
Feb  8 00:45:34 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: PluginManager: Monitor - No Plugins Changes observed! [4->4]
Feb  8 00:45:36 VHOSAKOT-M-H6X5 defaults[8426]: 
	The domain/default pair of (/Library/Preferences/com.jamfsoftware.jamf, global_log_level) does not exist
Feb  8 00:45:41 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Script "/opt/cisco/anyconnect/script/OnConnect_Mac_AnyConnect_Screenlock_D13v2.sh" exited with code 0.
Feb  8 00:45:50 VHOSAKOT-M-H6X5 diskimages-helper[8520]: *** -[NSMachPort handlePortMessage:]: dropping incoming DO message because the connection is invalid
Feb  8 00:45:51 VHOSAKOT-M-H6X5 com.apple.xpc.launchd[1] (com.apple.ftp-proxy): Unknown key for Boolean: ForceEnableHack
Feb  8 00:45:51 VHOSAKOT-M-H6X5 com.apple.xpc.launchd[1] (com.apple.bootpd): Unknown key for Boolean: ForceEnableHack
Feb  8 00:45:51 VHOSAKOT-M-H6X5 kernel[0]: bridge100: attached with 1 suspended link-layer multicast membership(s)
Feb  8 00:45:51 VHOSAKOT-M-H6X5 kernel[0]: en5: promiscuous mode enable succeeded
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: BCAST is ready [anyExternal, mtu=1406 ]
Feb  8 00:45:51 VHOSAKOT-M-H6X5 kernel[0]: bridge100: successfully restored 1 suspended link-layer multicast membership(s) (err=0)
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: added addr=192.168.64.1 mask=255.255.255.0 on bridge100
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: A new network interface has been detected.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: logInterfaces File: ../../vpn/AgentUtilities/Routing/InterfaceRouteMonitorCommon.cpp Line: 477 IP Address Interface List: FE80:0:0:0:CA69:CDFF:FEA0:4C88 192.168.0.29 2601:18B:4100:9505:CA69:CDFF:FEA0:4C88 2601:18B:4100:9505:E1CF:A829:66A:2A7A FE80:0:0:0:50D4:61FF:FE50:A62A 10.86.247.79 FE80:0:0:0:CA69:CDFF:FEA0:4C88 2001:420:C0E4:1002:0:0:0:AC 192.168.64.1 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Reconfigure reason code 15: New network interface.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: netInterfaceNoticeCategoryHandler File: ../../vpn/Agent/MainThread.cpp Line: 7965 Network Interface change detected, refreshing physical MAC addresses
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: A routing table change notification has been received.  Starting automatic correction of the routing table.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: started: [DHCP subnet=192.168.64/24 on bridge100 mtu=1500 <---> anyExternal mtu=1406] max-mss=1366
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]:   dns: 192.168.64.1
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: findMatchingRouteChange File: ../../vpn/AgentUtilities/Routing/RouteHandlerCommon.cpp Line: 4300 Found matching non-LL IPv4 VA default route.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: WaitWhileProcessingEvents File: ../../vpn/Agent/MainThread.cpp Line: 9794 Invoked Function: CMainThread::internalProcessEvents Return Code: -32702455 (0xFE0D0009) Description: MAINTHREAD_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: attempted to start dns proxy on anyExternal
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: docker-machine-driver-xhyve: com.apple.NetworkSharing.broadcast-0 has been started
Feb  8 00:45:51 VHOSAKOT-M-H6X5 mDNSResponder[109]: SetupDNSProxySkts: 14, 20, 25, 27
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: setDefaultRouteViaSysConfig File: ../../vpn/AgentUtilities/Routing/RouteHandlerMac.cpp Line: 605 Failed to get primary service IPv4 information
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: specialDefaultRouteAutoCorrect File: ../../vpn/AgentUtilities/Routing/RouteHandlerMac.cpp Line: 382 Invoked Function: CRouteHandlerMac::setDefaultRouteViaSysConfig Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: VerifyRouteTable File: ../../vpn/AgentUtilities/Routing/RouteHandlerCommon.cpp Line: 1049 Invoked Function: CRouteHandlerCommon::specialDefaultRouteAutoCorrect Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table - Original     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0    10.86.247.79  255.255.255.255     10.86.247.79     10.86.247.79                                                           utun0       9  N       0       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     169.254.0.0      255.255.0.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     192.168.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 255.255.255.255  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table - Modified     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric         0.0.0.0          0.0.0.0     10.86.247.79     10.86.247.79                                                           utun0       9  N       0         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0         0.0.0.0          0.0.0.0          0.0.0.0     192.168.64.1                                                       bridge100      11  Y       0    10.86.247.79  255.255.255.255          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     169.254.0.0      255.255.0.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     192.168.0.0    255.255.255.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     192.168.0.1  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.64.0    255.255.255.0          0.0.0.0     192.168.64.1                                                       bridge100      11  Y       0   198.135.0.166  255.255.255.255      192.168.0.1     192.168.0.29                                                             en0       4  N       0       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29      
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Index of questionable route entry in 'Modified' table: 2
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table changes: index  Action  Found     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric     0     NOP      Y         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0     1     DEL      N     192.168.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     2     ADD      N   198.135.0.166  255.255.255.255      192.168.0.1     192.168.0.29                                                             en0       4  N       0     3     ADD      N     192.168.0.0    255.255.255.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     4     NOP      Y         0.0.0.0          0.0.0.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     5     NOP      N 255.255.255.255  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     6     NOP      N       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     7     NOP      N    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     8     NOP      N       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     9     NOP      N       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnRouteTableChange File: ../../vpn/AgentUtilities/Routing/RouteMgr.cpp Line: 478 Invoked Function: IRouteHandler::VerifyRouteTable Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 com.apple.pfd[8524]: clearing states for internet-sharingshared_v4
Feb  8 00:45:51 VHOSAKOT-M-H6X5 InternetSharing[8523]: dns proxy successfully enabled
Feb  8 00:45:51 VHOSAKOT-M-H6X5 configd[60]: network changed: v4(en0:192.168.0.29) v6(utun0:2001:420:c0e4:1002::ac, en0) DNS! Proxy SMB
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database lists search domains: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: ; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 75.75.75.75, 75.75.76.76, 2001:558:feed::1, 2001:558:feed::2; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 161.44.124.122, 64.102.6.247; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database has domain name: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: WaitWhileProcessingEvents File: ../../vpn/Agent/MainThread.cpp Line: 9794 Invoked Function: CMainThread::internalProcessEvents Return Code: -32702455 (0xFE0D0009) Description: MAINTHREAD_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[814]: updating resolvers to nameserver 2001:558:feed::2#53
	timeout 2000
	order 200000
	nameserver 2001:558:feed::1#53
	timeout 2000
	order 200000
	nameserver 161.44.124.122#53
	timeout 2000
	order 200000
	nameserver 75.75.76.76#53
	timeout 2000
	order 200000
	nameserver 75.75.75.75#53
	timeout 2000
	order 200000
	nameserver 64.102.6.247#53
	timeout 2000
	order 200000
	search cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 configd[60]: network changed: v4(utun0+:10.86.247.79, en0) v6(utun0:2001:420:c0e4:1002::ac, en0) DNS! Proxy SMB
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database lists search domains: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 161.44.124.122, 64.102.6.247; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 75.75.75.75, 75.75.76.76, 2001:558:feed::1, 2001:558:feed::2; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 161.44.124.122, 64.102.6.247; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: WaitWhileProcessingEvents File: ../../vpn/Agent/MainThread.cpp Line: 9794 Invoked Function: CMainThread::internalProcessEvents Return Code: -32702455 (0xFE0D0009) Description: MAINTHREAD_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: executeRouteCmd File: ../../vpn/AgentUtilities/Routing/RouteTableMac.cpp Line: 219 route cmd success: route delete - dest 192.168.64.0/24, defGw 0.0.0.0, intf bridge100 (idx 11), metric 0, link-level
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table - fixed - deleted route     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric    192.168.64.0    255.255.255.0          0.0.0.0     192.168.64.1                                                       bridge100      11  Y       0 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database has domain name: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: WaitWhileProcessingEvents File: ../../vpn/Agent/MainThread.cpp Line: 9794 Invoked Function: CMainThread::internalProcessEvents Return Code: -32702455 (0xFE0D0009) Description: MAINTHREAD_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 configd[60]: network changed: v4(en0:192.168.0.29) v6(utun0:2001:420:c0e4:1002::ac, en0) DNS! Proxy SMB
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[814]: updating resolvers to nameserver 2001:558:feed::2#53
	timeout 2000
	order 200000
	nameserver 2001:558:feed::1#53
	timeout 2000
	order 200000
	nameserver 161.44.124.122#53
	timeout 2000
	order 200000
	nameserver 75.75.76.76#53
	timeout 2000
	order 200000
	nameserver 75.75.75.75#53
	timeout 2000
	order 200000
	nameserver 64.102.6.247#53
	timeout 2000
	order 200000
	search cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 com.apple.pfd[8524]: clearing states for internet-sharingshared_v4
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: setDefaultRouteViaSysConfig File: ../../vpn/AgentUtilities/Routing/RouteHandlerMac.cpp Line: 605 Failed to get primary service IPv4 information
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: specialVADefaultRouteAutoCorrect File: ../../vpn/AgentUtilities/Routing/RouteHandlerMac.cpp Line: 449 Invoked Function: CRouteHandlerMac::setDefaultRouteViaSysConfig Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: VerifyRouteTable File: ../../vpn/AgentUtilities/Routing/RouteHandlerCommon.cpp Line: 1197 Invoked Function: CRouteHandlerCommon::specialVADefaultRouteAutoCorrect Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table - Original     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0    10.86.247.79  255.255.255.255     10.86.247.79     10.86.247.79                                                           utun0       9  N       0       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     169.254.0.0      255.255.0.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     192.168.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 255.255.255.255  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table - Modified     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0         0.0.0.0          0.0.0.0          0.0.0.0     192.168.64.1                                                       bridge100      11  Y       0       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     169.254.0.0      255.255.0.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     192.168.0.0    255.255.255.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     192.168.0.1  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0    192.168.64.0    255.255.255.0          0.0.0.0     192.168.64.1                                                       bridge100      11  Y       0   198.135.0.166  255.255.255.255      192.168.0.1     192.168.0.29                                                             en0       4  N       0       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 255.255.255.255  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Routing table changes: index  Action  Found     Destination          Netmask          Gateway           IfAddr                                                          IfName IfIndex LL  Metric     0     NOP      Y         0.0.0.0          0.0.0.0      192.168.0.1     192.168.0.29                                                             en0       4  N       0     1     DEL      N     192.168.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     2     ADD      Y   198.135.0.166  255.255.255.255      192.168.0.1     192.168.0.29                                                             en0       4  N       0     3     ADD      Y     192.168.0.0    255.255.255.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     4     NOP      N         0.0.0.0          0.0.0.0          0.0.0.0     10.86.247.79                                                           utun0       9  Y       0     5     NOP      Y 255.255.255.255  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     6     NOP      Y       224.0.0.0    255.255.255.0          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     7     NOP      Y    192.168.0.29  255.255.255.255          0.0.0.0     192.168.0.29                                                             en0       4  Y       0     8     NOP      Y       127.0.0.1  255.255.255.255        127.0.0.1        127.0.0.1                                                             lo0       1  N       0     9     NOP      Y       127.0.0.0        255.0.0.0        127.0.0.1        127.0.0.1                                                             lo0       1  N       0    10     DEL      N    192.168.64.0    255.255.255.0          0.0.0.0     192.168.64.1                                                       b
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnRouteTableChange File: ../../vpn/AgentUtilities/Routing/RouteMgr.cpp Line: 478 Invoked Function: IRouteHandler::VerifyRouteTable Return Code: -24117239 (0xFE900009) Description: ROUTETABLE_ERROR_UNEXPECTED 
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database lists search domains: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: ; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 75.75.75.75, 75.75.76.76, 2001:558:feed::1, 2001:558:feed::2; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database includes DNS service: { Addresses: 161.44.124.122, 64.102.6.247; Order: 200000; Zones:  }
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Docker[816]: SC database has domain name: cisco.com
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Automatic correction of the routing table has failed.  Notifying higher levels of the routing change notification for possible further corrective action.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Reconfigure reason code 5: IP forwarding table modification.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: The entire VPN connection is being reconfigured.
Feb  8 00:45:51 VHOSAKOT-M-H6X5 acvpnagent[55]: Function: OnTunnelStateChange File: ../../vpn/Agent/TND.cpp Line: 1970 tunnel state change notification (new 2, old 1)
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: VPN state: Reconnecting Network state: Network Accessible Network control state: Network Access: Restricted Network type: Untrusted
Feb  8 00:45:51 VHOSAKOT-M-H6X5 Cisco AnyConnect Secure Mobility Client[8211]: Message type information sent to the user: Reconnecting to Boxborough - SSL...
@lowsky

This comment has been minimized.

Copy link

lowsky commented Feb 9, 2017

I found a solution for our VPN situation:

All the traffic is routed through the VPN tunnel, but there is a proxy for "browsing the internet".
In global VPN / network setting, there is a global.pac file ...

The static IP address of the proxy host is e.g. 12.34.56.78

Then I can set these environment to get kubectl working:

export https_proxy=http://12.34.56.78:8080
# optionally, for other tools like e.g. git
# export http_proxy=http://12.34.56.78:8080
# export ftp_proxy=http://12.34.56.78:8080
# export no_proxy=192.168.99.100,127.0.0.1

Hope this helps, at least after adapting for your specific proxy.

@duncanphillips

This comment has been minimized.

Copy link

duncanphillips commented Mar 8, 2017

is there no solution for this that works at the tcp/ip layer?

@duncanphillips

This comment has been minimized.

Copy link

duncanphillips commented Mar 8, 2017

for anyone else that come across this, there is a discussion on xhyve (machyve/xhyve#84), and the solution worked for me:

https://gist.github.com/mowings/633a16372fb30ee652336c8417091222

@blockloop

This comment has been minimized.

Copy link

blockloop commented Mar 8, 2017

@duncanphillips thanks. I'll give that a try. It's important to note that – according to machyve/xhyve#84 – you have to run this script every time the VPN starts

@nrichardson-nm

This comment has been minimized.

Copy link

nrichardson-nm commented Mar 22, 2017

Just posting here to help anyone else who has this problem.
This works if you're using a VPN with Secure Pulse (formerly Juniper Pulse I think) on macOS
Just run the following command AFTER connecting to your VPN (it must be run everytime you reconnect to your VPN)

sudo route -nv delete -net 192.168.99.0/24 -interface vboxnet0 (Only if you've ran the below command before)
sudo route -nv add -net 192.168.99.0/24 -interface vboxnet0

@blockloop

This comment has been minimized.

Copy link

blockloop commented Mar 23, 2017

@nrichardson-nm I am getting "File Exists" error with that one.

$ sudo route -nv add -net 192.168.99.0/24 -interface vboxnet0
u: inet 192.168.99.0; u: link vboxnet0:a.0.27.0.0.0; u: inet 255.255.255.0; RTM_ADD: Add Route: len 140, pid: 0, seq 1, errno 0, flags:<UP,STATIC>
locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK>
 192.168.99.0 vboxnet0:a.0.27.0.0.0 255.255.255.0
route: writing to routing socket: File exists
add net 192.168.99.0: gateway vboxnet0: File exists
@nrichardson-nm

This comment has been minimized.

Copy link

nrichardson-nm commented Mar 23, 2017

Yep. You have to delete it first if you've ever ran it. (Using the top command first)

@spuranam

This comment has been minimized.

Copy link

spuranam commented Apr 12, 2017

@blockloop I ran into this issue as well on OSX Sierra today with Cisco AnyConnect Client. I was able to work around this issue, by using native OSX VPN client, additionally i have to configure a HostOnly network in VMware Fusion, since there currently no way to pass network information to minikube i had to manually switch the network while the VM boots.

@neilneely

This comment has been minimized.

Copy link

neilneely commented May 4, 2017

@blockloop - I use Cisco AnyConnect as well, and at least my setup AnyConnect routes 192.168.96/19 through it's tunnel - this is clobbering the default minikube network of 192.168.99/24. I tried using the --host-only-cidr flag to use a different network, but either that doesn't work - or I did it wrong in some way (I suspect the later).

However - I was able to get this working by manually editing .minikube/machines/minikube/config.json and setting the following:

    "IPAddress": "10.254.254.100",
    "HostOnlyCIDR": "10.254.254.1/24",

I left all other values in there alone - just modifying the above two entries. With that set, I can start and stop minikube, and start and stop the VPN client, and everything always continues to work. (For this setup I'm using VirtualBox on a mac)

Note: my choice of 10.254.254/24 network was completely arbitrary, feel free to use whatever you prefer, as long as it isn't something else (like anyconnect) isn't already laying claim to.

@blockloop

This comment has been minimized.

Copy link

blockloop commented May 4, 2017

@neilneely I got a new job and don't have AnyConnect anymore so I cannot test, but that looks legit.

@thegridman

This comment has been minimized.

Copy link

thegridman commented Jun 8, 2017

I know this is closed but I'm adding this comment to describe how we made this work:

  1. Set port forwarding for the minikube vm to forward port 8443 on 127.0.0.1 to port 8443 in the VM.

VBoxManage controlvm minikube natpf1 k8s-apiserver,tcp,127.0.0.1,8443,,8443

  1. Create a new kubectl context
kubectl config set-cluster minikube-vpn --server=https://127.0.0.1:8443 --insecure-skip-tls-verify
kubectl config set-context minikube-vpn --cluster=minikube-vpn --user=minikube

When on the VPN you can set kubectl to use the NAT'ed port:

kubectl config use-context minikube-vpn

When off the VPN you can use the normal minikube context:

kubectl config use-context minikube

Apparently you can use the same port forwarding for the MiniKube dashboard too

VBoxManage controlvm minikube natpf1 k8s-dashboard,tcp,127.0.0.1,30000,,30000

@ClayShentrup

This comment has been minimized.

Copy link

ClayShentrup commented Jan 3, 2019

@thegridman Awesome solution! However, I spent most of today frustrated that minikube start was still hanging, like so:

I0102 22:28:05.612160   32960 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.99.121:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: dial tcp 192.168.99.121:8443: i/o timeout]
I0102 22:28:06.109164   32960 round_trippers.go:383] GET https://192.168.99.121:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy

Apparently that's fine. You just have to kill it, then run kubectl config use-context minikube again, and everything's fine. I don't know if there's anything less confusing I could instruct my team to do in our application README. 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment