Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter pod can't start; jinja exception Encountered unknown tag 'trans'. #325

Closed
jlewi opened this issue Mar 1, 2018 · 11 comments
Closed

Jupyter pod can't start; jinja exception Encountered unknown tag 'trans'. #325

jlewi opened this issue Mar 1, 2018 · 11 comments

Comments

@jlewi
Copy link
Contributor

jlewi commented Mar 1, 2018

I tried spawning a notebook with the Jupyter notebook image
gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-pr317

The pod appears to start and is killed right away. Looks like Jupyter reaps the pod right away making it difficult to debug.

I managed to run kubectl describe pods while it was still running and this is what I saw.

kubectl describe pods jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg 
Name:         jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg
Namespace:    kubeflow
Node:         gke-gpu-cluster-large-cpu-7e8d79a1-tt1w/10.128.0.4
Start Time:   Thu, 01 Mar 2018 14:59:33 -0800
Labels:       app=jupyterhub
              component=singleuser-server
              heritage=jupyterhub
              hub.jupyter.org/username=accounts_2Egoogle_2Ecom_3Ajlewi_40kubeflow_2Eorg
Annotations:  <none>
Status:       Running
IP:           10.32.0.63
Containers:
  notebook:
    Container ID:  docker://7c59dbf49d9bdb47242648114c85eb3a0ed455e0d35e8954dd74fcd783b01072
    Image:         gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-pr317
    Image ID:      docker-pullable://gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu@sha256:674f2dfebb7677f3c47705470b3f1e13e798ccb0a47d1883f6d85db30cb018ab
    Port:          8888/TCP
    Args:
      start-singleuser.sh
      --ip="0.0.0.0"
      --port=8888
      --allow-root
    State:          Running
      Started:      Thu, 01 Mar 2018 14:59:34 -0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      JUPYTERHUB_API_TOKEN:           eba01f3e073c49b68fc9476d596ee4e7
      JPY_API_TOKEN:                  eba01f3e073c49b68fc9476d596ee4e7
      JUPYTERHUB_CLIENT_ID:           user-accounts.google.com%3Ajlewi%40kubeflow.org
      JUPYTERHUB_HOST:                
      JUPYTERHUB_OAUTH_CALLBACK_URL:  /user/accounts.google.com%3Ajlewi@kubeflow.org/oauth_callback
      JUPYTERHUB_USER:                accounts.google.com:jlewi@kubeflow.org
      JUPYTERHUB_API_URL:             http://tf-hub-0:8081/hub/api
      JUPYTERHUB_BASE_URL:            /
      JUPYTERHUB_SERVICE_PREFIX:      /user/accounts.google.com%3Ajlewi@kubeflow.org/
      MEM_GUARANTEE:                  1Gi
      CPU_GUARANTEE:                  500m
    Mounts:
      /mnt/github-issues-data from github-issues-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from no-api-access-please (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  github-issues-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  github-issues-data
    ReadOnly:   false
  no-api-access-please:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                              Message
  ----    ------                 ----  ----                                              -------
  Normal  Scheduled              32s   default-scheduler                                 Successfully assigned jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg to gke-gpu-cluster-large-cpu-7e8d79a1-tt1w
  Normal  SuccessfulMountVolume  32s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  MountVolume.SetUp succeeded for volume "no-api-access-please"
  Normal  SuccessfulMountVolume  32s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  MountVolume.SetUp succeeded for volume "pvc-bfbfc706-1371-11e8-b633-42010a800335"
  Normal  Pulled                 31s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Container image "gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-pr317" already present on machine
  Normal  Created                31s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Created container
  Normal  Started                31s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Started container
jlewi@jlewi-carbon-glaptop:~/git_kubernetes-test-infra$ kubectl describe pods jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg 
Name:                      jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg
Namespace:                 kubeflow
Node:                      gke-gpu-cluster-large-cpu-7e8d79a1-tt1w/10.128.0.4
Start Time:                Thu, 01 Mar 2018 14:59:33 -0800
Labels:                    app=jupyterhub
                           component=singleuser-server
                           heritage=jupyterhub
                           hub.jupyter.org/username=accounts_2Egoogle_2Ecom_3Ajlewi_40kubeflow_2Eorg
Annotations:               <none>
Status:                    Terminating (expires Thu, 01 Mar 2018 15:00:08 -0800)
Termination Grace Period:  1s
IP:                        10.32.0.63
Containers:
  notebook:
    Container ID:  docker://7c59dbf49d9bdb47242648114c85eb3a0ed455e0d35e8954dd74fcd783b01072
    Image:         gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-pr317
    Image ID:      docker-pullable://gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu@sha256:674f2dfebb7677f3c47705470b3f1e13e798ccb0a47d1883f6d85db30cb018ab
    Port:          8888/TCP
    Args:
      start-singleuser.sh
      --ip="0.0.0.0"
      --port=8888
      --allow-root
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Thu, 01 Mar 2018 14:59:34 -0800
      Finished:     Thu, 01 Mar 2018 15:00:07 -0800
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      JUPYTERHUB_API_TOKEN:           eba01f3e073c49b68fc9476d596ee4e7
      JPY_API_TOKEN:                  eba01f3e073c49b68fc9476d596ee4e7
      JUPYTERHUB_CLIENT_ID:           user-accounts.google.com%3Ajlewi%40kubeflow.org
      JUPYTERHUB_HOST:                
      JUPYTERHUB_OAUTH_CALLBACK_URL:  /user/accounts.google.com%3Ajlewi@kubeflow.org/oauth_callback
      JUPYTERHUB_USER:                accounts.google.com:jlewi@kubeflow.org
      JUPYTERHUB_API_URL:             http://tf-hub-0:8081/hub/api
      JUPYTERHUB_BASE_URL:            /
      JUPYTERHUB_SERVICE_PREFIX:      /user/accounts.google.com%3Ajlewi@kubeflow.org/
      MEM_GUARANTEE:                  1Gi
      CPU_GUARANTEE:                  500m
    Mounts:
      /mnt/github-issues-data from github-issues-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from no-api-access-please (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  github-issues-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  github-issues-data
    ReadOnly:   false
  no-api-access-please:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                              Message
  ----    ------                 ----  ----                                              -------
  Normal  Scheduled              34s   default-scheduler                                 Successfully assigned jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg to gke-gpu-cluster-large-cpu-7e8d79a1-tt1w
  Normal  SuccessfulMountVolume  34s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  MountVolume.SetUp succeeded for volume "no-api-access-please"
  Normal  SuccessfulMountVolume  34s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  MountVolume.SetUp succeeded for volume "pvc-bfbfc706-1371-11e8-b633-42010a800335"
  Normal  Pulled                 33s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Container image "gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-pr317" already present on machine
  Normal  Created                33s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Created container
  Normal  Started                33s   kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Started container
  Normal  Killing                0s    kubelet, gke-gpu-cluster-large-cpu-7e8d79a1-tt1w  Killing container with id docker://notebook:Need to kill Pod
jlewi@jlewi-carbon-glaptop:~/git_kubernetes-test-infra$ kubectl logs jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg 
Error from server (NotFound): pods "jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg" not found

Looks like Jupyter exited as soon as it started.
@jlewi
Copy link
Contributor Author

jlewi commented Mar 1, 2018

In stack driver I used the following filter to get the logs

resource.type="container"
resource.labels.cluster_name="gpu-cluster"
resource.labels.namespace_id="kubeflow"
resource.labels.pod_id= "jupyter-accounts-2egoogle-2ecom-3ajlewi-40kubeflow-2eorg"

I see a jinja error

E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
 
E          return getattr(self, 'parse_' + self.stream.current.value)()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 268, in parse_block
 
E          node.body = self.parse_statements(('name:endblock',), drop_needle=True)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 165, in parse_statements
 
E          result = self.subparse(end_tokens)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
 
E          return getattr(self, 'parse_' + self.stream.current.value)()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 268, in parse_block
 
E          node.body = self.parse_statements(('name:endblock',), drop_needle=True)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 165, in parse_statements
 
E          result = self.subparse(end_tokens)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
 
E          return getattr(self, 'parse_' + self.stream.current.value)()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 213, in parse_if
 
E          'name:endif'))
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 165, in parse_statements
 
E          result = self.subparse(end_tokens)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 144, in parse_statement
 
E          self.fail_unknown_tag(token.value, token.lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 97, in fail_unknown_tag
 
E          return self._fail_ut_eof(name, self._end_token_stack, lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 90, in _fail_ut_eof
 
E          self.fail(' '.join(message), lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 59, in fail
 
E          raise exc(msg, lineno, self.name, self.filename)
 
E      jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'trans'. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.
 
E  [E 2018-03-01 22:48:10.453 SingleUserNotebookApp log:114] {
 
  undefined
E        "X-Forwarded-Host": "dev.kubeflow.org",
 
E        "X-Forwarded-Port": "80",
 
E        "X-Envoy-Expected-Rq-Timeout-Ms": "10000",
 
E        "Content-Length": "0",
 
E        "X-Envoy-Original-Path": "/user/accounts.google.com%3Ajlewi@kubeflow.org/tree?",
 
E        "X-Request-Id": "ca6c85f4-08b8-44c3-9eff-3e277b821f07",
 
E        "X-Forwarded-Proto": "https,http",
 
E        "X-Forwarded-For": "104.133.8.85, 35.190.59.219,10.32.5.29",
 
E        "Via": "1.1 google",
 
E        "Cookie": "user-accounts.google.com%3Ajlewi%40kubeflow.org=2|1:0|10:1519738241|47:user-accounts.google.com%3Ajlewi%40kubeflow.org|48:MWMxOWU4OWYtZmFjNS00ZmFhLTk4MjctOTk2YTNlOTU5ZDBh|66672d174b85a3cc6f4ddb53b1dcfbbf9b628d8d60c512bb879abd09561a8a63; _xsrf=2|17d97263|37c5377582d03dfa8054d273515cdfb2|1519738242; GCP_IAAP_XSRF_NONCE=Qo9AaULRF_IL9G2TmRJJdFtHBhNruc0dMq4ahPcqacc8k77mZy6s5kw6Mh56Xcifu54p_nuhQr_LZBwPxL0OyPIEeUcgO-Co3yWaJFoNimJfZef4ObQsZeSb1SCf-gxdXMC1j44q7AtN9ajdz2ZUv2oaLHcBXSKl3E1OVqY537w",
 
E        "X-Goog-Iap-Jwt-Assertion": "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik9rRXVHQSJ9.eyJpc3MiOiJodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vaWFwIiwic3ViIjoiYWNjb3VudHMuZ29vZ2xlLmNvbToxMTUwNjMyODE0NTYzMDMxMTk0NDIiLCJlbWFpbCI6ImpsZXdpQGt1YmVmbG93Lm9yZyIsImF1ZCI6Ii9wcm9qZWN0cy8yMzUwMzc1MDI5NjcvZ2xvYmFsL2JhY2tlbmRTZXJ2aWNlcy82NzYwMDgwNjc4NDUyOTA5MjY4IiwiZXhwIjoxNTE5OTQ1MDg5LCJpYXQiOjE1MTk5NDQ0ODksImhkIjoia3ViZWZsb3cub3JnIn0.I07Cnp-JrgXl0jnfyUYJP2ReBldyhqlh-nPxDI8hB8IoLrCOmUMrC8rzZ4eEcgAlp6Cfd8klViqa82wSDdF8gg",
 
E        "X-Goog-Authenticated-User-Email": "accounts.google.com:jlewi@kubeflow.org",
 
E        "X-Goog-Authenticated-User-Id": "accounts.google.com:115063281456303119442",
 
E        "X-Cloud-Trace-Context": "53faf9d15c4e7f856364368310bcf8c2/13504729293934828312",
 
E        "Accept-Language": "en-US,en;q=0.9",
 
E        "Accept-Encoding": "gzip, deflate, br",
 
E        "Referer": "https://dev.kubeflow.org/hub/home",
 
E        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
 
E        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",
 
E        "Upgrade-Insecure-Requests": "1",
 
E        "Host": "dev.kubeflow.org",
 
E        "Connection": "close"
 
E      }
 
E  [E 2018-03-01 22:48:10.453 SingleUserNotebookApp log:122] 500 GET /user/accounts.google.com%3Ajlewi@kubeflow.org/tree? (accounts.google.com:jlewi@kubeflow.org@10.32.5.29) 375.49ms
 
E  [E 2018-03-01 22:48:18.142 SingleUserNotebookApp web:1591] Uncaught exception GET /user/accounts.google.com%3Ajlewi@kubeflow.org/tree? (10.32.4.32)
 
E      HTTPServerRequest(protocol='http', host='dev.kubeflow.org', method='GET', uri='/user/accounts.google.com%3Ajlewi@kubeflow.org/tree?', version='HTTP/1.1', remote_ip='10.32.4.32', headers={'X-Forwarded-Host': 'dev.kubeflow.org', 'X-Forwarded-Port': '80', 'X-Envoy-Expected-Rq-Timeout-Ms': '10000', 'Content-Length': '0', 'X-Envoy-Original-Path': '/user/accounts.google.com%3Ajlewi@kubeflow.org/tree?', 'X-Request-Id': '03fccbdc-2db2-4c14-b530-de2ca12c51e1', 'X-Forwarded-Proto': 'https,http', 'X-Forwarded-For': '104.133.8.85, 35.190.59.219,10.32.4.32', 'Via': '1.1 google', 'Cookie': 'user-accounts.google.com%3Ajlewi%40kubeflow.org=2|1:0|10:1519738241|47:user-accounts.google.com%3Ajlewi%40kubeflow.org|48:MWMxOWU4OWYtZmFjNS00ZmFhLTk4MjctOTk2YTNlOTU5ZDBh|66672d174b85a3cc6f4ddb53b1dcfbbf9b628d8d60c512bb879abd09561a8a63; _xsrf=2|17d97263|37c5377582d03dfa8054d273515cdfb2|1519738242; GCP_IAAP_XSRF_NONCE=Qo9AaULRF_IL9G2TmRJJdFtHBhNruc0dMq4ahPcqacc8k77mZy6s5kw6Mh56Xcifu54p_nuhQr_LZBwPxL0OyPIEeUcgO-Co3yWaJFoNimJfZef4ObQsZeSb1SCf-gxdXMC1j44q7AtN9ajdz2ZUv2oaLHcBXSKl3E1OVqY537w', 'X-Goog-Iap-Jwt-Assertion': 'eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik9rRXVHQSJ9.eyJpc3MiOiJodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vaWFwIiwic3ViIjoiYWNjb3VudHMuZ29vZ2xlLmNvbToxMTUwNjMyODE0NTYzMDMxMTk0NDIiLCJlbWFpbCI6ImpsZXdpQGt1YmVmbG93Lm9yZyIsImF1ZCI6Ii9wcm9qZWN0cy8yMzUwMzc1MDI5NjcvZ2xvYmFsL2JhY2tlbmRTZXJ2aWNlcy82NzYwMDgwNjc4NDUyOTA5MjY4IiwiZXhwIjoxNTE5OTQ1MDk4LCJpYXQiOjE1MTk5NDQ0OTgsImhkIjoia3ViZWZsb3cub3JnIn0.aX929aGQLtnDkZemklj7oVVhk-DXCNJdl3_0uGqKVW5xDpeZkaiucIQMKXfEXx2H1aWBAx5OcpH2d0tCVnvx3Q', 'X-Goog-Authenticated-User-Email': 'accounts.google.com:jlewi@kubeflow.org', 'X-Goog-Authenticated-User-Id': 'accounts.google.com:115063281456303119442', 'X-Cloud-Trace-Context': '4894f7fe45aa171cb8deb85548261af5/6101762035122825447', 'Accept-Language': 'en-US,en;q=0.9', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36', 'Upgrade-Insecure-Requests': '1', 'Cache-Control': 'max-age=0', 'Host': 'dev.kubeflow.org', 'Connection': 'close'})
 
E      Traceback (most recent call last):
 
E        File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1510, in _execute
 
E          result = method(*self.path_args, **self.path_kwargs)
 
E        File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 2898, in wrapper
 
E          return method(self, *args, **kwargs)
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/tree/handlers.py", line 53, in get
 
E          server_root=self.settings['server_root_dir'],
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/base/handlers.py", line 347, in render_template
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/base/handlers.py", line 343, in get_template
 
E          .format(
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 830, in get_template
 
E          return self._load_template(name, self.make_globals(globals))
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 804, in _load_template
 
E          template = self.loader.load(self, name, globals)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/loaders.py", line 405, in load
 
E          return loader.load(environment, name, globals)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/loaders.py", line 125, in load
 
E          code = environment.compile(source, name, filename)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 591, in compile
 
E          self.handle_exception(exc_info, source_hint=source_hint)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
 
E          reraise(exc_type, exc_value, tb)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
 
E          raise value.with_traceback(tb)
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/templates/tree.html", line 23, in template
 
E          <li class="active"><a href="#notebooks" data-toggle="tab">{% trans %}Files{% endtrans %}</a></li>
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 497, in _parse
 
E          return Parser(self, source, name, encode_filename(filename)).parse()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 901, in parse
 
E          result = nodes.Template(self.subparse(), lineno=1)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
 
E          return getattr(self, 'parse_' + self.stream.current.value)()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 268, in parse_block
 
E          node.body = self.parse_statements(('name:endblock',), drop_needle=True)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 165, in parse_statements
 
E          result = self.subparse(end_tokens)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 144, in parse_statement
 
E          self.fail_unknown_tag(token.value, token.lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 97, in fail_unknown_tag
 
E          return self._fail_ut_eof(name, self._end_token_stack, lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 90, in _fail_ut_eof
 
E          self.fail(' '.join(message), lineno)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 59, in fail
 
E          raise exc(msg, lineno, self.name, self.filename)
 
E      jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'trans'. Jinja was looking for the following tags: 'endblock'. The innermost block that needs to be closed is 'block'.

@jlewi
Copy link
Contributor Author

jlewi commented Mar 1, 2018 

E  [E 2018-03-01 22:48:10.319 SingleUserNotebookApp web:1591] Uncaught exception GET /user/accounts.google.com%3Ajlewi@kubeflow.org/tree? (10.32.5.29)
 
E      HTTPServerRequest(protocol='http', host='dev.kubeflow.org', method='GET', uri='/user/accounts.google.com%3Ajlewi@kubeflow.org/tree?', version='HTTP/1.1', remote_ip='10.32.5.29', headers={'X-Forwarded-Host': 'dev.kubeflow.org', 'X-Forwarded-Port': '80', 'X-Envoy-Expected-Rq-Timeout-Ms': '10000', 'Content-Length': '0', 'X-Envoy-Original-Path': '/user/accounts.google.com%3Ajlewi@kubeflow.org/tree?', 'X-Request-Id': 'ca6c85f4-08b8-44c3-9eff-3e277b821f07', 'X-Forwarded-Proto': 'https,http', 'X-Forwarded-For': '104.133.8.85, 35.190.59.219,10.32.5.29', 'Via': '1.1 google', 'Cookie': 'user-accounts.google.com%3Ajlewi%40kubeflow.org=2|1:0|10:1519738241|47:user-accounts.google.com%3Ajlewi%40kubeflow.org|48:MWMxOWU4OWYtZmFjNS00ZmFhLTk4MjctOTk2YTNlOTU5ZDBh|66672d174b85a3cc6f4ddb53b1dcfbbf9b628d8d60c512bb879abd09561a8a63; _xsrf=2|17d97263|37c5377582d03dfa8054d273515cdfb2|1519738242; GCP_IAAP_XSRF_NONCE=Qo9AaULRF_IL9G2TmRJJdFtHBhNruc0dMq4ahPcqacc8k77mZy6s5kw6Mh56Xcifu54p_nuhQr_LZBwPxL0OyPIEeUcgO-Co3yWaJFoNimJfZef4ObQsZeSb1SCf-gxdXMC1j44q7AtN9ajdz2ZUv2oaLHcBXSKl3E1OVqY537w', 'X-Goog-Iap-Jwt-Assertion': 'eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik9rRXVHQSJ9.eyJpc3MiOiJodHRwczovL2Nsb3VkLmdvb2dsZS5jb20vaWFwIiwic3ViIjoiYWNjb3VudHMuZ29vZ2xlLmNvbToxMTUwNjMyODE0NTYzMDMxMTk0NDIiLCJlbWFpbCI6ImpsZXdpQGt1YmVmbG93Lm9yZyIsImF1ZCI6Ii9wcm9qZWN0cy8yMzUwMzc1MDI5NjcvZ2xvYmFsL2JhY2tlbmRTZXJ2aWNlcy82NzYwMDgwNjc4NDUyOTA5MjY4IiwiZXhwIjoxNTE5OTQ1MDg5LCJpYXQiOjE1MTk5NDQ0ODksImhkIjoia3ViZWZsb3cub3JnIn0.I07Cnp-JrgXl0jnfyUYJP2ReBldyhqlh-nPxDI8hB8IoLrCOmUMrC8rzZ4eEcgAlp6Cfd8klViqa82wSDdF8gg', 'X-Goog-Authenticated-User-Email': 'accounts.google.com:jlewi@kubeflow.org', 'X-Goog-Authenticated-User-Id': 'accounts.google.com:115063281456303119442', 'X-Cloud-Trace-Context': '53faf9d15c4e7f856364368310bcf8c2/13504729293934828312', 'Accept-Language': 'en-US,en;q=0.9', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://dev.kubeflow.org/hub/home', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36', 'Upgrade-Insecure-Requests': '1', 'Host': 'dev.kubeflow.org', 'Connection': 'close'})
 
E      Traceback (most recent call last):
 
E        File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1510, in _execute
 
E          result = method(*self.path_args, **self.path_kwargs)
 
E        File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 2898, in wrapper
 
E          return method(self, *args, **kwargs)
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/tree/handlers.py", line 53, in get
 
E          server_root=self.settings['server_root_dir'],
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/base/handlers.py", line 347, in render_template
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/base/handlers.py", line 343, in get_template
 
E          .format(
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 830, in get_template
 
E          return self._load_template(name, self.make_globals(globals))
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 804, in _load_template
 
E          template = self.loader.load(self, name, globals)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/loaders.py", line 405, in load
 
E          return loader.load(environment, name, globals)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/loaders.py", line 125, in load
 
E          code = environment.compile(source, name, filename)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 591, in compile
 
E          self.handle_exception(exc_info, source_hint=source_hint)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
 
E          reraise(exc_type, exc_value, tb)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
 
E          raise value.with_traceback(tb)
 
E        File "/opt/conda/lib/python3.6/site-packages/notebook/templates/tree.html", line 23, in template
 
E          <li class="active"><a href="#notebooks" data-toggle="tab">{% trans %}Files{% endtrans %}</a></li>
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 497, in _parse
 
E          return Parser(self, source, name, encode_filename(filename)).parse()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 901, in parse
 
E          result = nodes.Template(self.subparse(), lineno=1)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
 
E          return getattr(self, 'parse_' + self.stream.current.value)()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 268, in parse_block
 
E          node.body = self.parse_statements(('name:endblock',), drop_needle=True)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 165, in parse_statements
 
E          result = self.subparse(end_tokens)
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
 
E          rv = self.parse_statement()
 
E        File "/opt/conda/lib/python3.6/site-packages/jinja2/parser.py", line 144, in parse_statement

@jlewi
Copy link
Contributor Author

jlewi commented Mar 1, 2018

Here's the super relavant bit

Jinja is giving the following error

jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'trans'. Jinja was looking for the following tags: 'endblock'. The innermost block that needs to be closed is 'block'.

which appears to be coming from this piece of Jupyter code

 File "/opt/conda/lib/python3.6/site-packages/notebook/templates/tree.html", line 23, in template 
<li class="active"><a href="#notebooks" data-toggle="tab">{% trans %}Files{% endtrans %}</a></li>
 File "/opt/conda/lib/python3.6/site-packages/jinja2/environment.py", line 497, in _parse

@jlewi
Copy link
Contributor Author

jlewi commented Mar 1, 2018

It looks like this was added ~8 months ago as part of preliminary I18n support to Jupyter
jupyter/notebook@f81fb46

/cc @JCEmmons

@jlewi jlewi changed the title Jupyter pod killed as soon as it is spawned Jupyter pod can't start; jinja exception Encountered unknown tag 'trans'. Mar 2, 2018
@jlewi
Copy link
Contributor Author

jlewi commented Mar 2, 2018

/cc @ojarjur

@ojarjur
Copy link
Contributor

ojarjur commented Mar 2, 2018

I've been hitting this too. My Docker builds from yesterday were picking up version 5.0.0 of the notebook package, but today they've started picking up version 5.4.0.

Right now I'm experimenting with pinning to different versions to see if the problem will go away.

@jlewi
Copy link
Contributor Author

jlewi commented Mar 2, 2018

When I look at the files in the image /opt/conda/lib/python3.6/site-packages/notebook/templates I don't see any trans tags.

@jlewi
Copy link
Contributor Author

jlewi commented Mar 2, 2018

I looked at the hub pod and I find trans tags in the file
/usr/local/lib/python3.6/site-packages/notebook/templates/tree.html

@yuvipanda
Copy link
Contributor

Notebook recently started being localizable, and this seems related. I suspect some installation setup somehow that has the template files from a newer version of notebook than the code. Are you mixing conda and pip for the notebook package? Conda / pip mixing feels like the most likely culprit

@jlewi
Copy link
Contributor Author

jlewi commented Mar 2, 2018

Yeah looks like we're mixing conda and pip
https://github.com/kubeflow/kubeflow/blob/master/components/tensorflow-notebook-image/Dockerfile.cpu

@jlewi
Copy link
Contributor Author

jlewi commented Mar 2, 2018

Rebuilt the image and the new one works;

gcr.io/kubeflow-images-staging/tensorflow-notebook-cpu:v20180301-1838-pr317-1932b26-f88d"

I think my previous build might not have been a clean build.

@jlewi jlewi closed this as completed Mar 2, 2018
This was referenced Mar 2, 2018
Closed
Closed
yanniszark pushed a commit to arrikto/kubeflow that referenced this issue Nov 1, 2019
@k8s-ci-robot
tests: ACTUAL is printed as EXPECTED and vice-versa (kubeflow#325)
e929d3e
elenzio9 pushed a commit to arrikto/kubeflow that referenced this issue Oct 31, 2022
@andreyvelich
Add @andreyvelich to kf-kcc members (kubeflow#325)
db62ae4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yuvipanda @jlewi @ojarjur