Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter Notebook - ProxyError "426 Upgrade Required" in proxy environment #5554

Closed
kim-sardine opened this issue Jan 26, 2021 · 3 comments · May be fixed by aliceUnhinged613/kubeflow#186, k8s-mcrr/kubeflow-fork#34, deepk2u/kubeflow#85 or ajesse11x/kubeflow#260
Closed

Jupyter Notebook - ProxyError "426 Upgrade Required" in proxy environment #5554

kim-sardine opened this issue Jan 26, 2021 · 3 comments · May be fixed by aliceUnhinged613/kubeflow#186, k8s-mcrr/kubeflow-fork#34, deepk2u/kubeflow#85 or ajesse11x/kubeflow#260
Labels
area/jupyter Issues related to Jupyter kind/bug
Projects
Notebooks WG

Comments

@kim-sardine
Copy link

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I've setup kubernetes and kubeflow on internal proxy environment.

and I created Jupyter notebook using kubeflow Notebook server with default image (gcr.io/kubeflow-images-public/tensorflow-1.15.2-notebook-cpu:1.0.0).

when I ran pip install flask --trusted-host=pypi.python.org --cert=CERT_PATH --proxy=PROXY_URL (flask - for example) on this notebook, I got error below.

$ pip install -v flask --trusted-host=pypi.python.org --cert=CERT_PATH --proxy=PROXY_URL
Created temporary directory: /tmp/pip-ephem-wheel-cache-l7tdinso
Created temporary directory: /tmp/pip-req-tracker-yx6twg8u
Created requirements tracker '/tmp/pip-req-tracker-yx6twg8u'
Created temporary directory: /tmp/pip-install-di1nzvxp
Collecting flask
  1 location(s) to search for versions of flask:
  * https://pypi.org/simple/flask/
  Getting page https://pypi.org/simple/flask/
  Looking up "https://pypi.org/simple/flask/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  Incremented Retry for (url='/simple/flask/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 426 Upgrade Required',))': /simple/flask/

when I run same command on same image without kubeflow notebook server, I can install python packages.

$ docker run -p 8888:8888 --rm gcr.io/kubeflow-images-public/tensorflow-1.15.2-notebook-cpu:1.0.0

// Inside notebook terminal
$ pip install flask --trusted-host=pypi.python.org --proxy=PROXY_URL --cert=CERT_PATH
Collecting flask
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
     |████████████████████████████████| 94 kB 2.5 MB/s
Collecting itsdangerous>=0.24
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Requirement already sati

I've never seen 426 Upgrade Required error while working on proxy environment. I think this probably have something to do with jupyter-web-app or notebook-controller.

Environment:

  • Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): kfctl_istio_dex.v1.2.0
  • kfctl version: (use kfctl version): 1.2.0
  • Kubernetes platform: (e.g. minikube): kubeadm
  • Kubernetes version: (use kubectl version): v1.16.15
  • OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
@kubeflow-bot kubeflow-bot added this to To Do in Needs Triage Jan 26, 2021
@davidspek davidspek added this to Web Apps Stream in Notebooks WG Jan 26, 2021
@davidspek
Copy link
Contributor

@kim-sardine I think this is related to the notebook image you are using and not the Jupyter Web App or the notebook controller. Do you encounter this issue when using a different notebook image? You can try public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter:master-9d688f73 which is one of the new notebook images (this one does not have tensorflow installed).

@davidspek
Copy link
Contributor

/area jupyter

@google-oss-robot google-oss-robot added the area/jupyter Issues related to Jupyter label Mar 20, 2021
@kim-sardine
Copy link
Author

kim-sardine commented Mar 22, 2021 •
edited
Loading

@davidspek Thanks for your reply, same problem occurred with the image you said.

however, I've just found that it's an istio-proxy related issue.

since every jupyter notebook pod has istio-proxy sidecar container and I'm in internal enterprise proxy environment, istio-proxy have to know internal proxy server's information.

so I've created istio ServiceEntry (refer to this)

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
  name: internal-proxy
spec:
  addresses:
    - <PROXY_SERVER_IP>/32
  hosts:
    - internal.proxy # ignored anyway
  location: MESH_EXTERNAL
  ports:
    - name: http
      number: <PROXY_SERVER_PORT>
      protocol: TCP
  resolution: NONE

and it works like a charm.

Needs Triage automation moved this from To Do to Closed Mar 22, 2021
Notebooks WG automation moved this from Web Apps Stream to Done Mar 22, 2021
@kubeflow-bot kubeflow-bot removed this from Closed in Needs Triage Mar 22, 2021
@aliceUnhinged613 aliceUnhinged613 mentioned this issue Mar 26, 2024
Open
@k8s-mcrr k8s-mcrr mentioned this issue Mar 26, 2024
Open
@deepk2u deepk2u mentioned this issue Mar 27, 2024
Open
@ajesse11x ajesse11x mentioned this issue Mar 27, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/jupyter Issues related to Jupyter kind/bug
Projects
Notebooks WG
Done
Milestone
No milestone
4 participants
@kim-sardine @k8s-ci-robot @davidspek @google-oss-robot