New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container networking to cluster timeout #340

Closed
JoelSpeed opened this Issue Oct 30, 2017 · 1 comment

Comments

@JoelSpeed

JoelSpeed commented Oct 30, 2017

What were you trying to do?

Start a container, replacing a container in a deployment on a remote Kubernetes cluster (using --docker-run)

What did you expect to happen?

Container should start up and work as if running in the cluster

What happened instead?

Proxy connection hit timeout and exited with code 1. Seems container networking has an issue.

I tried to run the proxy container manually after the error came up and it returned code 100 which I believe is the expected output.

Automatically included information

Command line: ['/usr/local/bin/telepresence', '--swap-deployment', 'marvin', '--namespace', 'marvin', '--docker-run', '-v', '/var/run:/var/run', 'marvin']
Version: 0.68
Python version: 3.6.3 (default, Oct 4 2017, 06:09:38) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]
kubectl version: Client Version: v1.7.0
oc version: (error: [Errno 2] No such file or directory: 'oc': 'oc')
OS: Darwin Joels-MacBook-Pro.local 17.0.0 Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64 x86_64
Traceback:

Traceback (most recent call last):
  File "/usr/local/bin/telepresence", line 257, in call_f
    return f(*args, **kwargs)
  File "/usr/local/bin/telepresence", line 2379, in go
    ssh,
  File "/usr/local/bin/telepresence", line 2007, in run_docker_command
    TELEPRESENCE_LOCAL_IMAGE, "wait"
  File "/usr/local/bin/telepresence", line 384, in check_call
    raise CalledProcessError(retcode, args)
subprocess.CalledProcessError: Command '(['docker', 'run', '--network=container:telepresence-1509371680-5855958-13121', '--rm', 'datawire/telepresence-local:0.68', 'wait'],)' returned non-zero exit status 1.

Logs:

0 13:55:20 2017
  38.8 40 |    0.0 TL |   ['/usr/bin/entrypoint.py', 'proxy', '{"port": 64142, "cidrs": ["10.3.0.0/24"], "expose_ports": [], "ip": "198.18.0.254"}']
  38.8 41 | [INFO  tini (1)] Spawned child process 'python3' with pid '7'
  39.2 40 | Starting sshuttle proxy.
  39.7 40 | firewall manager: Starting firewall with Python version 3.6.1
  39.7 40 | firewall manager: ready method name nat.
  39.7 40 | IPv6 enabled: False
  39.7 40 | UDP enabled: False
  39.7 40 | DNS enabled: True
  39.7 40 | TCP redirector listening on ('127.0.0.1', 12300).
  39.7 40 | DNS listening on ('127.0.0.1', 12300).
  39.7 40 | Starting client with Python version 3.6.1
  39.7 40 | c : connecting to server...
  39.7 14 | Handling connection for 64142
  42.3 40 | Warning: Permanently added '[198.18.0.254]:64142' (ECDSA) to the list of known hosts.
  49.0 41 | Failed to connect to proxy in remote cluster.
  49.0 41 | [INFO  tini (1)] Main child exited normally (with status '1')
  49.3 TL | [41] exit 1.

@JoelSpeed

This comment has been minimized.

JoelSpeed commented Oct 30, 2017

I've rebuilt the telepresence-local upping the timeout to 30 instead of 10 and this has solved the problem.

Looking at the logs it's taking about 20 seconds for the wait to finish for me.

@ark3 ark3 self-assigned this Oct 30, 2017

@plombardi89 plombardi89 added this to Bugs in Roadmap Feb 21, 2018

@richarddli richarddli added this to Robustness in T Roadmap (v2) Feb 21, 2018

@rhs rhs added this to Error Feedback in Buckets Mar 8, 2018

Roadmap automation moved this from Bugs to Completed Mar 28, 2018

@ark3 ark3 closed this in a056d1f Mar 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment