How to login the Jupyterhub on the remote server?

[Smlight]

I have used kubeadm to setup my kubernetes, and used calico as the network extension.

[root@k8s-0001 ~]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                         READY     STATUS    RESTARTS   AGE
kube-system   calico-etcd-cgtcb                            1/1       Running   20         4d
kube-system   calico-kube-controllers-6ff88bf6d4-72s5k     1/1       Running   18         5d
kube-system   calico-node-bm9j5                            2/2       Running   1          5d
kube-system   calico-node-mt8vq                            2/2       Running   41         5d
kube-system   etcd-k8s-0001.novalocal                      1/1       Running   23         5d
kube-system   kube-apiserver-k8s-0001.novalocal            1/1       Running   23         5d
kube-system   kube-controller-manager-k8s-0001.novalocal   1/1       Running   25         5d
kube-system   kube-dns-545bc4bfd4-slfc4                    3/3       Running   54         5d
kube-system   kube-proxy-984hp                             1/1       Running   0          5d
kube-system   kube-proxy-xk6lw                             1/1       Running   20         5d
kube-system   kube-scheduler-k8s-0001.novalocal            1/1       Running   24         5d
kube-system   kubernetes-dashboard-747c4f7cf-6wvmp         1/1       Running   15         4d
kubeflow      ambassador-657699f59d-7n6tw                  1/2       Running   103        7h
kubeflow      ambassador-657699f59d-bptgn                  1/2       Running   108        7h
kubeflow      ambassador-657699f59d-xj6ch                  1/2       Running   103        7h
kubeflow      tf-hub-0                                     1/1       Running   0          7h
kubeflow      tf-job-operator-55c9bc9765-wl9dm             1/1       Running   0          3d

Since the service type LoadBalancer not work for my kubernetes, I have manually changed the type of tf-hub-lb to NodePort.

[root@k8s-0001 ~]# kubectl get svc --all-namespaces
NAMESPACE     NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes             ClusterIP   10.96.0.1        <none>        443/TCP         5d
kube-system   calico-etcd            ClusterIP   10.96.232.136    <none>        6666/TCP        5d
kube-system   kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   5d
kube-system   kubernetes-dashboard   NodePort    10.110.110.118   <none>        443:32200/TCP   5d
kube-system   tiller-deploy          ClusterIP   10.98.85.32      <none>        44134/TCP       1d
kubeflow      ambassador             ClusterIP   10.109.27.164    <none>        80/TCP          3d
kubeflow      ambassador-admin       ClusterIP   10.109.28.156    <none>        8877/TCP        3d
kubeflow      k8s-dashboard          ClusterIP   10.103.31.56     <none>        443/TCP         3d
kubeflow      tf-hub-0               ClusterIP   None             <none>        8000/TCP        3d
kubeflow      tf-hub-lb              NodePort    10.96.252.76     <none>        80:32133/TCP    3d

They are all on my remote server k8s-0001, suppose the public IP of it is a.b.c.d, here comes the problems:

1. I cannot visit a.b.c.d:32133 whether I go through http or https, only get a page said ERR_CONNECTION_TIMED_OUT, and the dashboard on port 32200 works well.
2. I try to access the jupyterhub directly. First, I added --no-ssl in the commands of the /root/my-kubeflow/vendor/kubeflow/core/jupyterhub.libsonnet. Second, I added c.Authenticator.admin_users = {'root'} at the end of the /root/my-kubeflow/vendor/kubeflow/core/jupyterhub_spawner.py. Third, I went into the pod by kubectl -n kubeflow exec tf-hub-0 -i -t -- bash -il and set a password for root. Fourth, I ran kubectl port-forward --namespace=kubeflow tf-hub-0 8888:8000 on my remote server. Fifth, I ran ssh -C -N -g -L 8888:127.0.0.1:8888 root@a.b.c.d on local machine. Then I could see the login page if I visit http://localhost:8888, but it turned to be ERR_EMPTY_RESPONSE when I hit the Sign In button. What's more, if I visit https://localhost:8888, I get a page said ERR_CONNECTION_CLOSED.

I just want to use Jupyterhub from public regradless of any security issue, help plz!!!

[jlewi]

It looks like there are two issues here

1. Connecting to JupyterHub
2. Authenticating to jupyter.

I suggest trying to tackle these issues separately; i.e lets figure out how to connect and JupyterHub and then lets figure out how to authenticate.

For connecting to JupyterHub

They are all on my remote server k8s-0001, suppose the public IP of it is a.b.c.d, here comes the problems:

I don't understand what you mean by "remote server"? Is this your K8s master or your K8s node? If you're using a node port then you need to use the IP address of one of your nodes and this IP address will need to be publicly accessible.

Did you try following the user guide and using port-forward to access Jupyter?

Why are you using port-forward to forward port 8888 to 8000 and ssh to also forward port 8888? Does this work? I would expect one of the commands to fail if you are already forwarding traffic on port 8888.

By default, JupyterHub uses the Dummy Authenticator which accepts any username and password.

Why are you changing the authenticator? Are you trying to set a password for JupyterHub?

Where and why are you adding "--no-ssl"?

[Smlight]

@jlewi Thank you and apologize, let me separate these.
Now I'm using all the default setting(not changing any command, service type, password, or authenticator)

NAMESPACE     NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubeflow      ambassador             ClusterIP   10.109.27.164    <none>        80/TCP          4d
kubeflow      ambassador-admin       ClusterIP   10.109.28.156    <none>        8877/TCP        4d
kubeflow      k8s-dashboard          ClusterIP   10.103.31.56     <none>        443/TCP         4d
kubeflow      tf-hub-0               ClusterIP   None             <none>        8000/TCP        4d
kubeflow      tf-hub-lb              ClusterIP   10.96.252.76     <none>        80/TCP          4d

They are running on k8s-0002.

NAME                 STATUS    ROLES     AGE       VERSION
k8s-0001.novalocal   Ready     master    6d        v1.8.1
k8s-0002.novalocal   Ready     <none>    6d        v1.8.1

If I run kubectl port-forward --namespace=kubeflow tf-hub-0 8000:8000 on master, I can get the web page by wget.

[root@k8s-0001 my-kubeflow]# wget 127.0.0.1:8000
--2018-03-06 19:54:35--  http://127.0.0.1:8000/
Connecting to 127.0.0.1:8000... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /hub [following]
--2018-03-06 19:54:35--  http://127.0.0.1:8000/hub
Reusing existing connection to 127.0.0.1:8000.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /hub/login [following]
--2018-03-06 19:54:35--  http://127.0.0.1:8000/hub/login
Reusing existing connection to 127.0.0.1:8000.
HTTP request sent, awaiting response... 200 OK
Length: 4565 (4.5K) [text/html]
Saving to: ‘index.html’

100%[====================================================================================================================>] 4,565       --.-K/s   in 0.04s

2018-03-06 19:54:35 (114 KB/s) - ‘index.html’ saved [4565/4565]

[root@k8s-0001 my-kubeflow]# cat index.html

<!DOCTYPE HTML>
<html>
…………………………
<div id="login-main" class="container">

<form action="/hub/login?next=" method="post" role="form">
  <div class="auth-form-header">
    Sign in
  </div>
  <div class='auth-form-body'>

    <p id='insecure-login-warning' class='hidden'>
    Warning: JupyterHub seems to be served over an unsecured HTTP connection.
    We strongly recommend enabling HTTPS for JupyterHub.
    </p>


    <label for="username_input">Username:</label>
    <input
      id="username_input"
      type="text"
      autocapitalize="off"
      autocorrect="off"
      class="form-control"
      name="username"
      val=""
      tabindex="1"
      autofocus="autofocus"
    />
    <label for='password_input'>Password:</label>
    <input
      type="password"
      class="form-control"
      name="password"
      id="password_input"
      tabindex="2"
    />

    <input
      type="submit"
      id="login_submit"
      class='btn btn-jupyter'
      value='Sign In'
      tabindex="3"
    />
  </div>
</form>

</div>
…………………………
</body>

It's fine, I had known this very well from the user guide. However, k8s-0002 cannot access it even they are in the same network, let alone public.

[root@k8s-0002 ~]# wget 192.168.0.187:8000
--2018-03-06 19:59:06--  http://192.168.0.187:8000/
Connecting to 192.168.0.187:8000... failed: Connection refused.

How can I connect to the jupyterhub from public? I'll follow your guide, thank you again!

[jlewi]

This appears to be an issue with your networking setup and not anything Kubeflow specific.

Have you been able to successfully expose a service the way you'd like e.g publicly? How did you do it? Are you running on a Cloud or in a cluster? Does your K8s distribution provide instructions about how to expose services publicly? Why doesn't your K8s distribution support LoadBalancer? Can you you use Ingress to direct traffic to the Jupyter service?

[root@k8s-0002 ~]# wget 192.168.0.187:8000
--2018-03-06 19:59:06--  http://192.168.0.187:8000/
Connecting to 192.168.0.187:8000... failed: Connection refused.

192.168.0.187 is not the pod IP. I'm guessing this is the ip address of k8s-0002. So I would not expect this to work. You could try setting the service type for JupyterHub to node port; this should cause it to be exposed on a port on the node.

[gpfvic]

@jlewi I came across the same problem as Smlight.
I deployed minikube on a server.
But I can't connect to the jupyterhub through my labtop browser.
I am new to the kubernetes and devops.

[jlewi]

@gpfvic Please open up a different issue. Your setup is likely very different since you are using minikube. If you are running minikube on a remote VM. You will need to figure out how to expose services running outside the VM. In the issue please describe what you've done to try to make this work.