Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Failed to check authorization" show on page when login jupyterhub after switch to dockerspawner #1566

Closed
ppLorins opened this issue Dec 1, 2017 · 11 comments

Comments

@ppLorins
Copy link

ppLorins commented Dec 1, 2017

errors
Hi all , I successfully lunched a jupyter-single-server inside a docker container by using docerspawner , when login into jupyterhub the page it shows me "Failed to check authorization" ,after check the container log I found the above errors , have no idea why it gave such errors , looks some token issue.

Anyone who ever experienced this ?

@minrk
Copy link
Member

minrk commented Dec 4, 2017

What are the logs from the Hub at the same time? Does the Hub also identify itself as version 0.8.1?

@ppLorins
Copy link
Author

ppLorins commented Dec 4, 2017

image

This is the log from the jupyterhub side. And jupyterhub version is 0.8.0 while jupyterhub-singleuser version is 0.8.0 , is the version difference the reason for my issue?

@minrk
Copy link
Member

minrk commented Dec 4, 2017

Fro, the Hub output, it's requesting /oauth2/token when it should be requesting /hub/api/oauth2/token. Something is amiss. This suggests that the JUPYTERHUB_API_URL environment variable may be incorrect. Can you docker inspect your container to see the environment variables? What is the version of dockerspawner?

@ppLorins
Copy link
Author

ppLorins commented Dec 4, 2017

[
    {
        "Id": "70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d",
        "Created": "2017-12-04T11:04:10.165772952Z",
        "Path": "jupyterhub-singleuser",
        "Args": [
            "--ip=\"0.0.0.0\"",
            "--port=8888",
            "--notebook-dir=\"/home/sad2/\"",
            "--hub-api-url=http://192.168.192.128:8081",
            "--allow-root",
            "--no-browser",
            ""
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 9622,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-12-04T11:04:10.374935354Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:ed06c00157039d299eec7c76c9e7f57a01f0f74a0ac3ba85b84beaf5fae9ac27",
        "ResolvConfPath": "/var/lib/docker/containers/70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d/hostname",
        "HostsPath": "/var/lib/docker/containers/70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d/hosts",
        "LogPath": "/var/lib/docker/containers/70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d/70ba9b0f996d841d6b568321b8e9809628aa9c7959493ab72aa3c29b3d95497d-json.log",
        "Name": "/jupyter-sad2",
        "RestartCount": 0,
        "Driver": "overlay",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/arthur/jupyterhub/jupyterhub-user-sad2:/home/sad2/:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "8888/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 52428800,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 104857600,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay/f7e7856b0bd32422ac3c572872c42f1096969087a6fe63e1406ecc3fd3ca1713/root",
                "MergedDir": "/var/lib/docker/overlay/da356a686b06d0a7b9fd2f11843d077f9d2e82aae89f9647092da969860fdd3d/merged",
                "UpperDir": "/var/lib/docker/overlay/da356a686b06d0a7b9fd2f11843d077f9d2e82aae89f9647092da969860fdd3d/upper",
                "WorkDir": "/var/lib/docker/overlay/da356a686b06d0a7b9fd2f11843d077f9d2e82aae89f9647092da969860fdd3d/work"
            },
            "Name": "overlay"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/arthur/jupyterhub/jupyterhub-user-sad2",
                "Destination": "/home/sad2",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "70ba9b0f996d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "8888/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "JUPYTERHUB_API_TOKEN=73ed60201dbc49df9b194e63dba1278b",
                "JPY_API_TOKEN=73ed60201dbc49df9b194e63dba1278b",
                "JUPYTERHUB_CLIENT_ID=user-sad2",
                "JUPYTERHUB_HOST=",
                "JUPYTERHUB_OAUTH_CALLBACK_URL=/user/sad2/oauth_callback",
                "JUPYTERHUB_USER=sad2",
                "JUPYTERHUB_API_URL=http://192.168.192.128:8081/hub/api",
                "JUPYTERHUB_BASE_URL=/",
                "JUPYTERHUB_SERVICE_PREFIX=/user/sad2/",
                "MEM_LIMIT=52428800",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "jupyterhub-singleuser",
                "--ip=\"0.0.0.0\"",
                "--port=8888",
                "--notebook-dir=\"/home/sad2/\"",
                "--hub-api-url=http://192.168.192.128:8081",
                "--allow-root",
                "--no-browser",
                ""
            ],
            "Image": "jupyter-single-server:done-v0",
            "Volumes": {
                "/home/sad2/": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "build-date": "20170911",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "52284f0650cf52873bf4fce8ff3f15eb28e54431ab3be7c34e1b3f1582a52cf3",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8888/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "32769"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/52284f0650cf",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "7fcd8bba396d244ad740bbb24d54874050758fee777f0dd866d9c7499e4d4c1b",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "49776a67e42d6cb5f2ef7dc36e88d84461a454cd35dad299ea2eea4cb21bf0c2",
                    "EndpointID": "7fcd8bba396d244ad740bbb24d54874050758fee777f0dd866d9c7499e4d4c1b",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

This is my docker container inspect xx output , and my docker spawner version is 0.9.1.
"JUPYTERHUB_API_URL=http://192.168.192.128:8081/hub/api", this url seems right.
Does the '/hub/api' prefix should also be added to somewhere else?

@ppLorins
Copy link
Author

ppLorins commented Dec 4, 2017

pip3 list of jupyterhub environment:

alembic (0.9.6)
bleach (2.1.1)
certifi (2017.7.27.1)
chardet (3.0.4)
decorator (4.1.2)
docker (2.6.1)
docker-pycreds (0.2.1)
dockerspawner (0.9.1)
entrypoints (0.2.3)
escapism (1.0.0)
html5lib (1.0b10)
idna (2.6)
ipykernel (4.6.1)
ipython (6.2.1)
ipython-genutils (0.2.0)
jedi (0.11.0)
Jinja2 (2.9.6)
jsonschema (2.6.0)
jupyter-client (5.1.0)
jupyter-core (4.4.0)
jupyterhub (0.8.0)
Mako (1.0.7)
MarkupSafe (1.0)
mistune (0.8)
nbconvert (5.3.1)
nbformat (4.4.0)
notebook (5.2.0)
pamela (0.3.0)
pandocfilters (1.4.2)
parso (0.1.0)
pexpect (4.2.1)
pickleshare (0.7.4)
pip (9.0.1)
prompt-toolkit (1.0.15)
ptyprocess (0.5.2)
Pygments (2.2.0)
python-dateutil (2.6.1)
python-editor (1.0.3)
python-oauth2 (1.0.1)
pyzmq (16.0.3)
requests (2.18.4)
setuptools (28.8.0)
simplegeneric (0.8.1)
six (1.11.0)
SQLAlchemy (1.1.14)
terminado (0.6)
testpath (0.3.1)
tornado (4.5.2)
traitlets (4.3.2)
urllib3 (1.22)
wcwidth (0.1.7)
webencodings (0.5.1)
websocket-client (0.44.0)

@minrk
Copy link
Member

minrk commented Dec 5, 2017

@ppLorins have you set c.DockerSpawner.hub_ip_connect? Can you share your whole configuration (redacted as appropriate)?

@minrk
Copy link
Member

minrk commented Dec 5, 2017

JUPYTERHUB_API_URL env is right, but the command being launched has --hub-api-url=http://192.168.192.128:8081 which is wrong and has higher priority. We need to identify whatever is setting this parameter and fix it.

@ppLorins
Copy link
Author

ppLorins commented Dec 7, 2017

import sys
import os

sys.path.append(os.path.dirname(__file__))

c.JupyterHub.api_tokens = { '2b744e22bf88470eb117399ac1ccc275':'arthur' }
c.JupyterHub.authenticator_class = 'my_auth_dir.my_auth.MyAuthenticator'
c.JupyterHub.hub_ip = '192.168.192.128'
c.JupyterHub.hub_port = 8081
c.ConfigurableHTTPProxy.api_url='http://192.168.192.128:8082'
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
notebook_dir =  '/home/{username}/'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes      = {'/home/arthur/jupyterhub/jupyterhub-user-{username}' :notebook_dir}
c.Spawner.mem_limit          = '50M'
c.DockerSpawner.image        = 'jupyter-single-server:done-v0'
c.JupyterHub.ssl_cert = '/home/arthur/certs/server.crt'
c.JupyterHub.ssl_key = '/home/arthur/certs/server.key'
c.Spawner.args = ['--hub-api-url=http://192.168.192.128:8081','--allow-root','--no-browser',\
                '']
c.Spawner.cmd = ['jupyterhub-singleuser']
c.Authenticator.admin_users = {'arthur'}
c.LocalAuthenticator.add_user_cmd = ['adduser', '--home','/home/USERNAME']
c.LocalAuthenticator.create_system_users = True
c.PAMAuthenticator.open_sessions = False

@minrk This is my jupyterhub_config.py with all comments removed.

@minrk
Copy link
Member

minrk commented Dec 8, 2017

It's this line that's the problem:

c.Spawner.args = ['--hub-api-url=http://192.168.192.128:8081','--allow-root','--no-browser',\
                '']

I would recommend removing this line altogether, because the value for hub-api-url is unnecessary and incorrect (the source of the error). --no-browser is also unnecessary, and --allow-root is hopefully unnecessary, as I hope your users aren't all running as root.

@ppLorins
Copy link
Author

@minrk Yes. The problem solved after I removed the hub-api-url option in the c.Spawner.args.
Thanks a lot.

@harmon
Copy link

harmon commented Apr 25, 2018

Sweet mimbalee bimbalee, this was my EXACT issue as well! I needed to set the API url to point internally in my docker network, but didn't realize that the path "/hub/api" was needed!

This saved me a bunch of time, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants