Error Running Hello World Example #178

BaadDaad · 2017-06-15T10:56:05Z

What were you trying to do?

Run the telepresence Hello World example

What did you expect to happen?

I expected it to match the documentation at http://www.telepresence.io/tutorials/openshift.html

What happened instead?

Automatically included information

Command line: ['/usr/local/bin/telepresence', '-m', 'inject-tcp', '--new-deployment', 'example', '--run', 'curl', 'http://myservice:8000/']
Version: 0.51
Python version: 3.6.1 (v3.6.1:69c0db5050, Mar 21 2017, 01:21:04) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
kubectl version: (error: [Errno 2] No such file or directory: 'kubectl')
oc version: `oc v1.5.1+7b451fc
kubernetes v1.5.2+43a9be4
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v1.5.1+7b451fc
kubernetes v1.5.2+43a9be4OS:Darwin MC02SY10KH03M 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64`
Traceback:

Traceback (most recent call last):
  File "/usr/local/bin/telepresence", line 95, in call_f
    return f(*args, **kwargs)
  File "/usr/local/bin/telepresence", line 1477, in go
    runner, args
  File "/usr/local/bin/telepresence", line 931, in start_proxy
    run_id=run_id,
  File "/usr/local/bin/telepresence", line 596, in get_remote_info
    wait_for_pod(runner, remote_info)
  File "/usr/local/bin/telepresence", line 673, in wait_for_pod
    "Pod isn't starting or can't be found: {}".format(pod["status"])
RuntimeError: Pod isn't starting or can't be found: {'conditions': [{'lastProbeTime': None, 'lastTransitionTime': '2017-06-15T10:51:16Z', 'status': 'True', 'type': 'Initialized'}, {'lastProbeTime': None, 'lastTransitionTime': '2017-06-15T10:51:16Z', 'message': 'containers with unready status: [example]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'Ready'}, {'lastProbeTime': None, 'lastTransitionTime': '2017-06-15T10:51:16Z', 'status': 'True', 'type': 'PodScheduled'}], 'containerStatuses': [{'image': 'datawire/telepresence-k8s:0.51', 'imageID': '', 'lastState': {}, 'name': 'example', 'ready': False, 'restartCount': 0, 'state': {'waiting': {'message': 'Back-off pulling image "datawire/telepresence-k8s:0.51"', 'reason': 'ImagePullBackOff'}}}], 'hostIP': '192.168.65.2', 'phase': 'Pending', 'podIP': '172.17.0.7', 'startTime': '2017-06-15T10:51:16Z'}

Logs:

namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!
Running: (['oc', '--context', 'myproject/127-0-0-1:8443/developer', '--namespace', 'myproject', 'get', 'pod', 'example-1-0vc82', '-o', 'json'],)... ran!

The text was updated successfully, but these errors were encountered:

itamarst · 2017-06-15T13:51:05Z

Hi,

Sorry this isn't working for you.

Which OpenShift environment are you using? (My experience with OpenShift is that it is often slow, an OpenShift Online free environment is extra slow. Also the latter basically only lets you run two-three services at a time, and additional services will never be scheduled.)
Could you try again and see if it works this time?

BaadDaad · 2017-06-15T14:22:52Z

Hi, Itamar,

Thanks for responding so quickly.

I'm not sure what you mean by 'Openshift environment'. I'm running Openshift Origin 1.5.1 version on my laptop.
Still not working.

Tony

itamarst · 2017-06-15T14:28:26Z

So looking in more detail the issue appears to be that it can't pull the image:

{'message': 'Back-off pulling image "datawire/telepresence-k8s:0.51"', 'reason': 'ImagePullBackOff'}

This image is hosted on the Docker Hub, so it ought to be accessible, it's listed in Docker Hub page.

Does this work?

$ oc run myservice --image=datawire/hello-world --port=8000 --expose

Work in this case means "after a minute you can run oc get pod and it will have a "myservice" pod in Running phase."

itamarst · 2017-06-15T14:29:46Z

BTW, if you want to debug this more interactively you can stop by the Gitter chat - https://gitter.im/datawire/telepresence

BaadDaad · 2017-06-15T14:46:54Z

Seems to be OK:

15:45 $ oc get pod
NAME READY STATUS RESTARTS AGE
myservice-1-9k07w 1/1 Running 0 21s

Unfortunately, Gitter chat is blocked by our proxy.

itamarst · 2017-06-15T14:55:35Z

Ok, how about if you do:

$ oc run test --image=datawire/telepresence-k8s:0.51

Does that eventually result in a pod? I can definitely pull that image on my computer.

BaadDaad · 2017-06-15T15:28:37Z

That doesn't seem to work:

hello-world-1-8ghcd 0/1 ImagePullBackOff 0 34m

itamarst · 2017-06-15T15:40:47Z

That is very confusing... what does oc describe pod hello-world-1-8ghcd (or whatever the latest pod name is) and oc describe dc hello-world say?

itamarst · 2017-06-15T15:46:15Z

Also, can you try directly with Docker? Will hopefully make it easier to see why image can't be pulled

$ docker pull datawire/telepresence-k8s:0.51

BaadDaad · 2017-06-15T15:59:00Z

Current pods are:

16:41 $ oc get pods
NAME                READY     STATUS             RESTARTS   AGE
myservice-1-sbb7b   1/1       Running            0          5m
test-1-deploy       1/1       Running            0          8m
test-1-k719k        0/1       ImagePullBackOff   0          8m

16:48 $ oc describe pod myservice-1-sbb7b
Name:			myservice-1-sbb7b
Namespace:		myproject
Security Policy:	restricted
Node:			192.168.65.2/192.168.65.2
Start Time:		Thu, 15 Jun 2017 16:48:17 +0100
Labels:			deployment=myservice-1
			deploymentconfig=myservice
			run=myservice
Status:			Running
IP:			172.17.0.7
Controllers:		ReplicationController/myservice-1
Containers:
  myservice:
    Container ID:	docker://a01de0c16b40730453cb40c0d80fab204d591ac3ea3c054d1f4a981917599c43
    Image:		datawire/hello-world
    Image ID:		docker-pullable://datawire/hello-world@sha256:e6e0ae04bbf45cda5ffca52ee7fc6a7f05e2604aee29e91db75a72d39194a84a
    Port:		8000/TCP
    State:		Running
      Started:		Thu, 15 Jun 2017 16:48:20 +0100
    Ready:		True
    Restart Count:	0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nvg5d (ro)
    Environment Variables:	<none>
Conditions:
  Type		Status
  Initialized 	True
  Ready 	True
  PodScheduled 	True
Volumes:
  default-token-nvg5d:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-nvg5d
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath			Type		Reason		Message
  ---------	--------	-----	----			-------------			--------	------		-------
  34s		34s		1	{default-scheduler }					Normal		Scheduled	Successfully assigned myservice-1-sbb7b to 192.168.65.2
  33s		33s		1	{kubelet 192.168.65.2}	spec.containers{myservice}	Normal		Pulling		pulling image "datawire/hello-world"
  31s		31s		1	{kubelet 192.168.65.2}	spec.containers{myservice}	Normal		Pulled		Successfully pulled image "datawire/hello-world"
  31s		31s		1	{kubelet 192.168.65.2}	spec.containers{myservice}	Normal		Created		Created container with docker id a01de0c16b40; Security:[seccomp=unconfined]
  31s		31s		1	{kubelet 192.168.65.2}	spec.containers{myservice}	Normal		Started		Started container with docker id a01de0c16b40

16:56 $ oc describe pod test-1-deploy
Name:			test-1-deploy
Namespace:		myproject
Security Policy:	restricted
Node:			192.168.65.2/192.168.65.2
Start Time:		Thu, 15 Jun 2017 16:45:37 +0100
Labels:			openshift.io/deployer-pod-for.name=test-1
Status:			Failed
IP:			172.17.0.2
Controllers:		<none>
Containers:
  deployment:
    Container ID:	docker://b9cbe83a2f97a558b0adbfb0a7e78910f226939a34065e745f821b54520d241a
    Image:		openshift/origin-deployer:v1.5.1
    Image ID:		docker-pullable://openshift/origin-deployer@sha256:77ac551235d8edf43ccb2fbd8fa5384ad9d8b94ba726f778fced18710c5f74f0
    Port:
    State:		Terminated
      Reason:		Error
      Exit Code:	1
      Started:		Thu, 15 Jun 2017 16:45:38 +0100
      Finished:		Thu, 15 Jun 2017 16:55:39 +0100
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from deployer-token-w3lq5 (ro)
    Environment Variables:
      KUBERNETES_MASTER:	https://192.168.65.2:8443
      OPENSHIFT_MASTER:		https://192.168.65.2:8443
      BEARER_TOKEN_FILE:	/var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:	-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE0OTY2Nzc3MzUwHhcNMTcwNjA1MTU0ODU0WhcNMjIwNjA0
MTU0ODU1WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE0OTY2Nzc3MzUw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDFQZ6BOfjkc6gap2vJZnjQ
6d7o9LqknNRTnWav/QFZ+i2czuVdp+OEI8+91QI00KZhYj2Nv2rDhB4RuAoqGvxO
BsfYbOp+pfIVtXt5cAtXaK5chZ2/Vv1n1+paZ2x5gJFwwLGwY6Mptlt50+yV3JwQ
CSC6rDhq46bhlvTEioCVTXRpiG0szCpJ3YjJUwckxOtE4r3RGilygNDLW/sNbxrE
G/U8flLKv/odxCKK3vie9zQ8IbhB/mRgiyOdoutNqRmLjitFcjO/X0cg3ocvykYQ
nAQOApXZRClIs4DLI0HffAMgjRifmG4/lfYQf07S0hLYmaHuj6Laai1GYBTsMx7v
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQCzNo2DVziGBm87SfgNxv0/phNdL/mNkOkd6Q7BZuc5KqTc
8YhFE3WzjzQ1FBSzR/LgIbvDbImA22EOtBYSI/ij96gQAeeHbKCxdh8sXgLt6CPX
bpVdBvbpzqL5y0iUsTLV+2n3lcwM9wuKNByKttmZeuVerJ49iBLWwm3whSPt4EOC
hk32UIL+amLMTkjMDM1EJXiTvcOnu2x5nHl4VFIJhuGRumv5L6cjF8vlEaHr+z2w
No6TVZNQzMvpA6PdmVaa09BBxRX7YS/FuPjnwBzPrxqmQHhSE5G8W23qpXHop5R9
0xj3xqDd6WKpkY3rjV7sMny920kOkiYLrslKFdob
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:	test-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:	myproject
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  deployer-token-w3lq5:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	deployer-token-w3lq5
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath			Type		Reason		Message
  ---------	--------	-----	----			-------------			--------	------		-------
  11m		11m		1	{default-scheduler }					Normal		Scheduled	Successfully assigned test-1-deploy to 192.168.65.2
  11m		11m		1	{kubelet 192.168.65.2}	spec.containers{deployment}	Normal		Pulled		Container image "openshift/origin-deployer:v1.5.1" already present on machine
  11m		11m		1	{kubelet 192.168.65.2}	spec.containers{deployment}	Normal		Created		Created container with docker id b9cbe83a2f97; Security:[seccomp=unconfined]
  11m		11m		1	{kubelet 192.168.65.2}	spec.containers{deployment}	Normal		Started		Started container with docker id b9cbe83a2f97

itamarst · 2017-06-15T16:32:39Z

What do you get for oc describe on the pod with ImagePullBackOff status?
What do you get for oc describe test?
What do you get for docker pull datawire/telepresence-k8s:0.51?

BaadDaad · 2017-06-16T08:06:38Z

First attempt to pull your image directly:

08:51 $ docker pull datawire/telepresence-k8s:0.51
Error response from daemon: Get https://registry-1.docker.io/v2/: x509: certificate has expired or is not yet valid

After adding registry-1.docker.io to my insecure registries and restarting docker:

✘-1 /tmp/telepresence-test
08:54 $ docker pull datawire/telepresence-k8s:0.51
0.51: Pulling from datawire/telepresence-k8s
6f821164d5b7: Pull complete
f867842595c0: Pull complete
39135a879327: Pull complete
78119914bd24: Pull complete
1e8e129bc1c6: Pull complete
13873a21d1d7: Pull complete
7c9cc2d6ce7f: Pull complete
0fde9b6b0f06: Pull complete
Digest: sha256:1dbb7b6796269e734c822d3a8b65d4f403f4264e369fabfb4b83a56cb95997df
Status: Downloaded newer image for datawire/telepresence-k8s:0.51
✔ /tmp/telepresence-test
09:04 $

itamarst · 2017-06-16T13:41:15Z

Thanks for your patience in debugging this.

A couple of reasons this might happen (I can think of more, but let's start with these):

Docker somehow broke their certificate temporarily. This happens sometimes. At the moment it looks valid to me, though.
The date on the machines running Docker and OpenShift does not match reality. This is fairly common, and in particular if you're on a Mac then Docker is in a VM, as would be OpenShift (I assume you're using minishift?).

If you're using minishift, can you ssh in with minishift ssh, and check the date? If it's wrong you can probably fix it by running:

$ ntpclient -s -h pool.ntp.org

Or otherwise change the date.

Likewise, if you're on a Mac you can restart Docker and see if that fixes issue like so: moby/moby#11534 (comment)

BaadDaad · 2017-06-16T15:25:13Z

I think it's almost there. Unfortunately I have another process bound to port 8000 and the proxy obviously fails when it starts up.
What changes would I need to make to use a different port?

BaadDaad · 2017-06-16T16:08:22Z

OK, Itamar, I have it working now.

Thanks for your help.

itamarst · 2017-06-16T17:25:10Z

Right now the port on your machine needs to match the port of the Docker image. For the example image it's hardcoded to 8000, so you can substitute a different image that listens on a different port. E.g. I think openshift/hello-openshift image listens on 8080.

Your own images you can change the port in the image to not calsh, but... that is a problem in general, yes.

The ideal would be to say someting like --expose 8001:8000, so it maps 8001 on your machine to 8000 in the deployment. I'll open a new ticket for that and CC you so you can see when it's done.

itamarst added the bug Something isn't working label Jun 15, 2017

BaadDaad closed this as completed Jun 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error Running Hello World Example #178

Error Running Hello World Example #178

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017 •

edited

Loading

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017 •

edited

Loading

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 16, 2017

itamarst commented Jun 16, 2017

BaadDaad commented Jun 16, 2017

BaadDaad commented Jun 16, 2017

itamarst commented Jun 16, 2017

Error Running Hello World Example #178

Error Running Hello World Example #178

Comments

BaadDaad commented Jun 15, 2017

What were you trying to do?

What did you expect to happen?

What happened instead?

Automatically included information

itamarst commented Jun 15, 2017 • edited Loading

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017 • edited Loading

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 15, 2017

itamarst commented Jun 15, 2017

BaadDaad commented Jun 16, 2017

itamarst commented Jun 16, 2017

BaadDaad commented Jun 16, 2017

BaadDaad commented Jun 16, 2017

itamarst commented Jun 16, 2017

itamarst commented Jun 15, 2017 •

edited

Loading

itamarst commented Jun 15, 2017 •

edited

Loading