400: Bad Request - OAuth state missing from cookies (Google auth) #1028

anton-khodak · 2018-11-19T16:52:19Z

Hi,

I want to use Google authentication with JupyterHub. The problem is that cookie "oauthenticator" that get set up by JupyterHub disappears once Google redirects from oauth page back to Jupyter. Thus I see

403 : Forbidden 
You are not signed in to your jupyter.cellgeni.sanger.ac.uk account.

and on refresh

400 : Bad Request
OAuth state missing from cookies

I noticed that among other cookies "oauthenticator" one is the only one set with HttpOnly: true. This is the only reason I can think of to be invalidated since the expiration date for the cookie is one day ahead.

This is the cookie state after the server sets up cookies :

Once Google authenticated a user and returns them back to JupyterHub, the oauthenticator cookie just disappears. Other cookies stay.

I looked into jupyterhub/jupyterhub#2044 and jupyterhub/jupyterhub#1519, didn't help to understand how to fix this problem.

Hope someone can give a clue how to handle it.

Contents of `config.yaml`

proxy:
  secretToken: <token>

auth:
  type: google
  google:
    clientId: "<client-id>"
    clientSecret: "<secret>"
    callbackUrl: "https://jupyter.cellgeni.sanger.ac.uk/hub/oauth_callback"
    hostedDomain: "jupyter.cellgeni.sanger.ac.uk"
    loginService: "JupyterHub at Wellcome Trust Sanger Institute"

cull:
  timeout: 129600

singleuser:
  defaultUrl: "/lab"
  storage:
    capacity: 30Gi
  memory:
    limit: 20G
    guarantee: 16G
  cpu:
    limit: 4
    guarantee: 2
  image:
    name: quay.io/cellgeni/cellgeni-jupyter
    tag: v0.2.8
  lifecycleHooks:
    postStart:
      exec:
        command: ["bash", "/poststart.sh"]

The text was updated successfully, but these errors were encountered:

divatemangesh · 2019-03-09T11:21:54Z

I am getting 500 Internal server error when autheticating with GITLAB-CE seems like authentication-state cookies expiration issue

cleared browser cookies and cache and any saved data and in New Tab opened https://localhost
security warning came like site is not safe I clicked on proceed safely
In chrome ctrl+shift+C>> application >> cookies I can see following entries
Sign in with gitlab page

Name:- xsrf
Value:- 2|d6b491b6|bf36d80df77faf9466583355e204eb77|1550066103
Expires:- 2020-03-15T13:55:03.000Z

Name:- username-localhost-8888
Value:-"2|1:0|10:1551431956|23:username-localhost-8888|44:MmU4ZDlmY2JkMzRkNDNjNWJhOTJiNmQ4NzNmYjA4NDA=|d5174b5df574432e334f4733f11943c12caf3cf3e05c4b47c7c962937c2a8960"
Expires:- 2020-03-31T09:19:16.000Z
-------------------------------- Once Clicked on Sign in with gitlab button ----------------------------

additional one entry came and disappered in less than half second and I landed to GitLab-CE login page but I screen recorded it to see the disappred entry

The one which disappeared was

Name :- oauthenticator-state
Value:- "2|1:0|10:1552127982|20:oauthenticator-state|120:Z... was not able to see comple value as it was partialy visiable in recording
expires :- 2019-03-10T1 ...

-------------------Once landed on gitlab-CE login-----------------------------------

Name:- _gitlab_session
Value:- ba46390addc98a11c22e4cd4dd064fdb
Expires:- 2019-03-09T12:39:42.873Z

---------------------loged in to gitlab with username-----------------------------
_gitlab_session
6dc3812c2eb2a0593ed8cb6c4f4f12b6
1969-12-31T23:59:59.000Z

--------------------Once Clicked on authosise in gitlab------------
Name:- _xsrf
Value :- 2|d6b491b6|bf36d80df77faf9466583355e204eb77|1550066103
Expireds:- 2020-03-15T13:55:03.000Z

GOT 500 INTERNAL SERVER ERROR

If issue is due authentication state cookies expiring before creating then how to solve it

abdidarmawan007 · 2019-03-20T06:40:31Z

this is epic fail bug never fix lol

consideRatio · 2019-03-20T07:58:55Z

Hmmm, cookies are valid in various domains. I recently learned about what HTTP=true cookies imply and that should be fine still. It makes the cookie inaccessible from javascript but is still passed when making HTTP requests from the browser/client when GET/POST-ing etc to a webserver.

I'm not confident about what goes on, but cookies are things on the browser being passed to the webserver when making requests etc, and depending on the domain, different cookies are sent. So if facebook.com stores a cookie with HTTP=true on the browser by returning a "Set-Cookie" response header, it will be sent back to the webserver when the browser makes the next request back to facebook.com, but only facebook.com

So, if you browse what cookies are available while on google.com, those relates to google.com, and google.com will certainly store various auth related cookies, but jupyterhub wants to store a separate one i assume.

Hmmm, questions:

what does it say in the hub logs when making the getting the 403 error?
what version of the helm chart are you using?
does upgrading to --version=0.9-8463734 resolve the issues? (Includes Switch to openssl backend for pycurl #1185 and Fix bug affecting config of authenticators #1171)
UPDATE: No, the issue is resolved in --version=0.9-9c580a1 though I think.

Info:

There is a bug fixed with google auth by @joshbode in Switch to openssl backend for pycurl #1185, I'm not at all confident this is the same issue though, but could perhaps be figured out by looking in the hub pod's logs.
I fixed some bugs in Fix bug affecting config of authenticators #1171 (that impact gitlab @divatemangesh, but not googles authenticator I think @anton-khodak)

nscozzaro · 2019-08-17T18:28:17Z

I'm also experiencing this issue:

consideRatio · 2019-08-17T19:03:27Z

Is there someone having a functional google authenticator up and running?

nscozzaro · 2019-08-17T20:12:25Z

Not sure if it's related, but while trying to debug this I ran helm upgrade ... again. My hub then started going into a crash loop, so to debug I ran kubectl logs hub-8576fdcd66-v599d (that's the name of my hub pod), and I see the following traceback:

>>> kubectl logs hub-8576fdcd66-v599d
Loading /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
[I 2019-08-17 20:04:39.343 JupyterHub app:1673] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-0.8.0
[I 2019-08-17 20:04:39.343 JupyterHub app:1673] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2019-08-17 20:04:39.344 JupyterHub app:1016] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[I 2019-08-17 20:04:39.362 JupyterHub dbutil:125] Upgrading sqlite:///jupyterhub.sqlite
[I 2019-08-17 20:04:39.363 JupyterHub dbutil:105] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2019-08-17-200439
[I 2019-08-17 20:04:39.972 alembic.runtime.migration migration:130] Context impl SQLiteImpl.
[I 2019-08-17 20:04:39.973 alembic.runtime.migration migration:137] Will assume non-transactional DDL.
FAILED: Can't locate revision identified by '4dc2d5a8c53c'
[E 2019-08-17 20:04:39.977 alembic.util.messaging messaging:60] Can't locate revision identified by '4dc2d5a8c53c'
[E 2019-08-17 20:04:40.092 JupyterHub app:1958]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1955, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1680, in initialize
        self.init_db()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1073, in init_db
        dbutil.upgrade_if_needed(self.db_url, log=self.log)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/dbutil.py", line 130, in upgrade_if_needed
        upgrade(db_url)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/dbutil.py", line 89, in upgrade
        ['alembic', '-c', alembic_ini, 'upgrade', revision]
      File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpvq9ylmnn/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 255.

nscozzaro · 2019-08-18T16:25:00Z

Is there someone having a functional google authenticator up and running?

@consideRatio, I now have a functional Google authenticator running.

What worked was to use the latest development release of the helm chart (--version 0.9-470ec04) and to remove "hostedDomain" setting for Google auth in the config.yaml

And, to solve the error I was getting above I added the following to config.yaml

hub:
  db:
    upgrade: true

which I tried because I saw a similar issue here: #1244

moki298 · 2019-08-23T15:39:37Z

Is there someone having a functional google authenticator up and running?

Yes I do

pbadenski · 2020-01-10T18:05:18Z

Is there plans to get this out in stable release version?

consideRatio · 2020-01-10T18:56:06Z

I dont have a clear idea about when this issue occur or why yet :/

kyleatmousera · 2020-01-31T23:07:04Z

I had this same error "400: Bad Request - OAuth state missing from cookies" when using (Google auth)
I found that removing "hostedDomain" setting for Google auth in the config.yaml worked for helm chart version 0.8.2.

csudanthi · 2020-04-13T10:37:21Z

had to remove hostedDomain as well also helm chart version 0.8.2.

consideRatio · 2020-10-06T00:42:03Z

I'll go ahead and close this issue for now, please open a new issue if this fails on 0.9.0+.

lakshaykc mentioned this issue Jul 2, 2019

GenericOAuthenticator error : HTTP 599: gnutls_handshake() failed #1321

Closed

consideRatio closed this as completed Oct 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

400: Bad Request - OAuth state missing from cookies (Google auth) #1028

400: Bad Request - OAuth state missing from cookies (Google auth) #1028

anton-khodak commented Nov 19, 2018 •

edited

Loading

divatemangesh commented Mar 9, 2019

abdidarmawan007 commented Mar 20, 2019

consideRatio commented Mar 20, 2019 •

edited

Loading

nscozzaro commented Aug 17, 2019

consideRatio commented Aug 17, 2019

nscozzaro commented Aug 17, 2019 •

edited

Loading

nscozzaro commented Aug 18, 2019 •

edited

Loading

moki298 commented Aug 23, 2019

pbadenski commented Jan 10, 2020

consideRatio commented Jan 10, 2020

kyleatmousera commented Jan 31, 2020 •

edited

Loading

csudanthi commented Apr 13, 2020

consideRatio commented Oct 6, 2020

400: Bad Request - OAuth state missing from cookies (Google auth) #1028

400: Bad Request - OAuth state missing from cookies (Google auth) #1028

Comments

anton-khodak commented Nov 19, 2018 • edited Loading

Contents of config.yaml

divatemangesh commented Mar 9, 2019

abdidarmawan007 commented Mar 20, 2019

consideRatio commented Mar 20, 2019 • edited Loading

nscozzaro commented Aug 17, 2019

consideRatio commented Aug 17, 2019

nscozzaro commented Aug 17, 2019 • edited Loading

nscozzaro commented Aug 18, 2019 • edited Loading

moki298 commented Aug 23, 2019

pbadenski commented Jan 10, 2020

consideRatio commented Jan 10, 2020

kyleatmousera commented Jan 31, 2020 • edited Loading

csudanthi commented Apr 13, 2020

consideRatio commented Oct 6, 2020

anton-khodak commented Nov 19, 2018 •

edited

Loading

Contents of `config.yaml`

consideRatio commented Mar 20, 2019 •

edited

Loading

nscozzaro commented Aug 17, 2019 •

edited

Loading

nscozzaro commented Aug 18, 2019 •

edited

Loading

kyleatmousera commented Jan 31, 2020 •

edited

Loading