Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Telepresence finishes cleanup before deployment config is restored [Openshift] #929

Closed
bartoszmajsak opened this Issue Feb 20, 2019 · 0 comments

Comments

Projects
None yet
1 participant
@bartoszmajsak
Copy link
Contributor

bartoszmajsak commented Feb 20, 2019

When finishing Telepresence session running on Openshift, deployment config is rolled back to its original state using oc rollout latest dc/name. This, however, is not blocking operation, so there's a chance that quickly restarting Telepresence session will result in error such as:

telepresence.log
   0.0 TEL | Telepresence 0.97-dirty launched at Tue Feb 19 22:38:53 2019
   0.0 TEL |   /usr/local/bin/telepresence --swap-deployment hello-world --expose 8000 --method inject-tcp --run python3 server.py
   0.0 TEL |   Using images version 0.97 (dev)
   0.0 TEL | Platform: linux
   0.0 TEL | Python 3.7.2 (default, Jan 16 2019, 19:49:22)
   0.0 TEL | [GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
   0.0 TEL | [1] Running: uname -a
   0.0   1 | Linux hydrobook 4.20.7-200.fc29.x86_64 #1 SMP Wed Feb 6 19:16:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
   0.0 TEL | BEGIN SPAN main.py:40(main)
   0.0 TEL | BEGIN SPAN startup.py:74(__init__)
   0.0 TEL | Found kubectl -> /home/bartek/.asdf/shims/kubectl
   0.0 TEL | Found oc -> /home/bartek/.asdf/shims/oc
   0.0 TEL | [2] Capturing: kubectl version --short
   0.2 TEL | [3] Capturing: kubectl config current-context
   0.3 TEL | [4] Capturing: kubectl config view -o json
   0.7 TEL | [5] Capturing: kubectl --context hello-world/127-0-0-1:8443/developer get ns hello-world
   1.0 TEL | Command: oc 1.11.6
   1.0 TEL | Context: hello-world/127-0-0-1:8443/developer, namespace: hello-world, version: 1.11.0+d4cacc0
   1.0 TEL | [6] Capturing: minishift ip
   1.1 TEL | END SPAN startup.py:74(__init__)    1.1s
   1.1 TEL | [7] Capturing: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world get dc/hello-world
   1.4   7 | NAME          REVISION   DESIRED   CURRENT   TRIGGERED BY
   1.4   7 | hello-world   28         1         0         config,image(hello-world:latest)
   1.4 TEL | Found ssh -> /usr/bin/ssh
   1.4 TEL | [8] Capturing: ssh -V
   1.4 TEL | Found python3 -> /usr/bin/python3
   1.4 TEL | Found torsocks -> /usr/bin/torsocks
   1.4 TEL | Found sshfs -> /usr/bin/sshfs
   1.4 TEL | Found fusermount -> /usr/bin/fusermount
   1.4 TEL | [9] Capturing: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world get pods telepresence-connectivity-check --ignore-not-found
   3.0 TEL | Scout info: {'latest_version': '0.97', 'application': 'telepresence', 'notices': []}
   3.0 TEL | [10] Capturing: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world get dc/hello-world -o json --export
   3.4 TEL | [11] Running: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world set triggers dc/hello-world --remove-all
   3.7  11 | deploymentconfig.apps.openshift.io/hello-world triggers updated
   3.7 TEL | [12] Capturing: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world get dc/hello-world -o json --export
   3.9 TEL | [13] Running: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world replace -f -
   4.1  13 | deploymentconfig.apps.openshift.io/hello-world replaced
   4.1 TEL | [14] Running: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world rollout latest dc/hello-world
   4.4  14 | error: #28 is already in progress (Running).
   4.4 TEL | [14] exit 1 in 0.25 secs.
   6.4 TEL | CRASH: Command '['oc', '--context', 'hello-world/127-0-0-1:8443/developer', '--namespace', 'hello-world', 'rollout', 'latest', 'dc/hello-world']' returned non-zero exit status 1.
   6.4 TEL | Traceback (most recent call last):
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/cli.py", line 130, in crash_reporting
   6.4 TEL |     yield
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/main.py", line 60, in main
   6.4 TEL |     remote_info = start_proxy(runner)
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/proxy/__init__.py", line 98, in start_proxy
   6.4 TEL |     runner_, deployment_arg, image_name, args.expose, add_custom_ns
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/proxy/deployment.py", line 374, in swap_deployment_openshift
   6.4 TEL |     apply_json(new_dc_json)
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/proxy/deployment.py", line 357, in apply_json
   6.4 TEL |     runner.kubectl("rollout", "latest", "dc/{}".format(deployment))
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/runner/runner.py", line 346, in check_call
   6.4 TEL |     track, "Running", "ran", out_cb, err_cb, args, **kwargs
   6.4 TEL |   File "/usr/local/bin/telepresence/telepresence/runner/runner.py", line 335, in _run_command
   6.4 TEL |     raise CalledProcessError(retcode, args)
   6.4 TEL | subprocess.CalledProcessError: Command '['oc', '--context', 'hello-world/127-0-0-1:8443/developer', '--namespace', 'hello-world', 'rollout', 'latest', 'dc/hello-world']' returned non-zero exit status 1.
   6.4 TEL | (calling crash reporter...)
   6.4 >>> | Exit cleanup in progress
   6.4 TEL | (Cleanup) Restore original deployment config
   6.4 TEL | [15] Running: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world replace -f -
   6.9  15 | deploymentconfig.apps.openshift.io/hello-world replaced
   6.9 TEL | [16] Running: oc --context hello-world/127-0-0-1:8443/developer --namespace hello-world rollout latest dc/hello-world
   7.1  16 | error: #28 is already in progress (Running).
   7.1 TEL | [16] exit 1 in 0.24 secs.
   7.1 TEL | (Cleanup) Restore original deployment config failed:
   7.1 TEL | (Cleanup)   Command '['oc', '--context', 'hello-world/127-0-0-1:8443/developer', '--namespace', 'hello-world', 'rollout', 'latest', 'dc/hello-world']' returned non-zero exit status 1.
   7.1 TEL | (Cleanup) Stop time tracking
   7.1 TEL | END SPAN main.py:40(main)    7.1s
   7.1 TEL | (Cleanup) Remove temporary directory
   7.1 TEL | (Cleanup) Save caches
Solution

Cleanup process should wait until old deployment config is restored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.