Resolve issue 128 - yubikey auth flow broken by google page update #136

adcreare · 2019-06-14T12:28:40Z

This makes an attempt to resolve issue #128
Works for me with a single yubikey on my google corporate account.

Google changed the id-challenge field to contain just the challenge txt, they include the other information now hidden away inside a div tag further down in a json like object that contains a bunch of nulls.
They also provide some of the data in base64 now but their api expects base64 url encoded on some of the fields, despite all that being base64'd itself.

This PR has resolved the issue for us, I'd be very keen to hear from others if this works for them.
Comments, feedback etc welcome, I'm not exactly a python expert so any points there welcome too :-)

- added functions to handle custom field object - handle challange id value being single - various base64 decoding + base64 url encoding now required by google on post payload

coveralls · 2019-06-14T20:44:30Z

Coverage decreased (-1.6%) to 46.756% when pulling aa4ab76 on adcreare:master into 6f5919d on cevoaustralia:master.

hadalin · 2019-06-17T09:40:12Z

Seems to be working for me 👍

You should probably add a test.

mhenniges · 2019-06-17T17:00:34Z

For some reason, this doesnt work for me with a yubikey 4, but I'm not sure exactly why yet, based on my limited knowledge it looks like the right fields are being sent. I'll try some more testing later and try spot the issue.

adcreare · 2019-06-17T21:16:55Z

@hadalin agreed re tests: I'd like to expand the tests a lot if I can so its easier in the future for others to understand the flow as well - took me awhile to get my head around what was going on

@mhenniges interesting - do you get an error of any kind?

I'll try update the tests later in the week

mhenniges · 2019-06-18T15:29:59Z

@adcreare , frustratingly, no, I don't get any error other than the final 'Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.' saml.html indeed does not have a saml response, its that same html page that you were seeing before this PR.

I'm not exactly sure what to look for to make additional progress on this. So far, I've done the following:
1 - added logging for pretty much every variable and http response in order to check that things are getting parsed and identified properly. I think that part is working fine; all the values look sane to me.
2 - added a print of json.dumps(u2f_challenges) in the handle_sk method in google.py; and validated that the printed json does successfully result in a signed challenge response when fed into ' u2f-authenticate https://accounts.google.com'. I did patch my u2f_host library to allow that facet, as the comments in u2f.py discuss.

3 - dropped the starting url (https://accounts.google.com/o/saml2/initsso...) into my browser, and captured a trace as I went through the flow. Everything worked fine in chrome, and I ended up on the AWS console. I then captured a trace of aws-google-auth trying to authenticate and compared the two.

In the post to submit the u2f challenge step, I see the following differences:

Present in the form data of browser trace but absent in aws-google-auth trace:
checkConnection: youtube:285:1
flowName: GlifWebSignIn

Fields gfx, TL, id-challenge, and id-assertion had different contents. Also, in the id-assertion, there was an additional field in the browser version: "sessionId":""

In the 'continue' field, the 'as' term was different, but in each it seemed consistent with the redirects that had come previously.

Also, the browser trace has a cookie called 'GAPS'.

Of course those differences may be completely normal...

I'm happy to do other tests if there's something specific I should try. I don't have any leads at all at the moment.

sinkr · 2019-06-19T19:23:48Z

Hmm, this does not work for me. I have verified that I have installed your PR and am invoking it, however, I still get this error after captcha verification (you'll notice I incremented the version to 0.0.32 just so I could verify I'm running the correct code):

ERROR:root:Object of type bytes is not JSON serializable Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/__init__.py", line 72, in cli process_auth(args, config) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/__init__.py", line 212, in process_auth google_client.do_login() File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/google.py", line 293, in do_login sess = self.handle_sk(sess) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/google.py", line 417, in handle_sk auth_response_dict = u2f.u2f_auth(u2f_challenges, facet) File "/usr/local/lib/python3.7/site-packages/aws_google_auth-0.0.32-py3.7.egg/aws_google_auth/u2f.py", line 61, in u2f_auth return u2f.authenticate(device, json.dumps(challenge), File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.__class__.__name__} ' TypeError: Object of type bytes is not JSON serializable

adcreare · 2019-06-19T21:22:27Z

very odd - i'm using it daily and it seems ok. We have a whole team who use it as well so I'll push this fix out to them and see how they go.

out of interest what version of python are you guys running, i'm running 2.7

sinkr · 2019-06-19T21:26:01Z

very odd - i'm using it daily and it seems ok. We have a whole team who use it as well so I'll push this fix out to them and see how they go.

out of interest what version of python are you guys running, i'm running 2.7

Interestingly enough, 3.7. I reinstalled it with 2.7 and don't get the above error, but I now get this which I see @mhenniges got above:

Touch the flashing U2F device to authenticate... Something went wrong - Could not find SAML response, check your credentials or use --save-failure-html to debug.

mhenniges · 2019-06-20T11:24:47Z

@adcreare - I'm running 2.7.16 as installed by brew on OSX 10.14.5, and my dependencies resolved as follows:

$ pipdeptree -p aws-google-auth,python-u2flib-host
aws-google-auth==0.0.31

beautifulsoup4 [required: Any, installed: 4.7.1]
- soupsieve [required: >=1.2, installed: 1.9.1]
  - backports.functools-lru-cache [required: Any, installed: 1.5]
boto3 [required: Any, installed: 1.9.168]
- botocore [required: >=1.12.168,<1.13.0, installed: 1.12.168]
  - docutils [required: >=0.10, installed: 0.14]
  - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
  - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
  - urllib3 [required: >=1.20,<1.26, installed: 1.25.3]
- jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
- s3transfer [required: >=0.2.0,<0.3.0, installed: 0.2.1]
  - botocore [required: >=1.12.36,<2.0.0, installed: 1.12.168]
    - docutils [required: >=0.10, installed: 0.14]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
    - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - urllib3 [required: >=1.20,<1.26, installed: 1.25.3]
  - futures [required: >=2.2.0,<4.0.0, installed: 3.2.0]
configparser [required: Any, installed: 3.7.4]
keyring [required: Any, installed: 18.0.1]
- entrypoints [required: Any, installed: 0.3]
  - configparser [required: >=3.5, installed: 3.7.4]
keyrings.alt [required: Any, installed: 3.1.1]
- six [required: Any, installed: 1.12.0]
lxml [required: Any, installed: 4.3.4]
Pillow [required: Any, installed: 6.0.0]
requests [required: Any, installed: 2.22.0]
- certifi [required: >=2017.4.17, installed: 2019.3.9]
- chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
- idna [required: >=2.5,<2.9, installed: 2.8]
- urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
six [required: Any, installed: 1.12.0]
tabulate [required: Any, installed: 0.8.3]
tzlocal [required: Any, installed: 1.5.1]
- pytz [required: Any, installed: 2019.1]

python-u2flib-host==3.0.3

hidapi [required: >=0.7.99, installed: 0.7.99.post21]
- setuptools [required: >=19.0, installed: 40.8.0]
requests [required: Any, installed: 2.22.0]
- certifi [required: >=2017.4.17, installed: 2019.3.9]
- chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
- idna [required: >=2.5,<2.9, installed: 2.8]
- urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]

how does this compare to your setup?

- fixes random login error

adcreare · 2019-06-20T23:07:35Z

I think i found the issue for the SAML not found - basically google wasn't always accepting the auth response when it includes base64 padding in the challenge response.

I've added something to remove the padding before its passed to the u2f code to sign and that seems to fix it.

base64 padding shouldn't (in theory) cause an issue but seems that it is in this case.
Give it a try see if it helps. I haven't had a chance to look into the python 3.x issue - I'll do that when I get a chance.

mhenniges · 2019-06-21T16:34:16Z

@adcreare nice work! I've got nothing but successes with the current version. Is this ready to merge then @stevemac007 ? Anything I can do to help push it over the finish line if not?

robertsosinski · 2019-06-21T20:47:21Z

Hey all,

So I got it almost working with pulling @adcreare master branch on Python 2.7, however when I get to the Yubikey part, I get this:

ERROR:root:No U2F device found. 5 attempts remaining
Insert your U2F device and press enter to try again...

No matter how many times I use my Yubikey, and I see the input streaming into the command prompt, it fails.

I'm using a Yubikey 5 NFC.

Thanks!

robertsosinski · 2019-06-21T21:34:53Z

Just wanted to chime in again. Confirmed this same issue on both Windows 10 cmd, and Ubuntu 18.04 Linux Subsystem on Windows 10.

Thanks!

adcreare · 2019-06-21T22:12:41Z

Just wanted to chime in again. Confirmed this same issue on both Windows 10 cmd, and Ubuntu 18.04 Linux Subsystem on Windows 10.

Thanks!

Haven't changed any of the U2F device detection stuff - I'd suggest making sure you have install the package with U2F support. We have had this issue internally on some macs our solution was to install https://www.python.org/ftp/python/2.7.15/python-2.7.15-macosx10.9.pkg and that cleared it up.

adcreare · 2019-06-21T22:14:18Z

@adcreare nice work! I've got nothing but successes with the current version. Is this ready to merge then @stevemac007 ? Anything I can do to help push it over the finish line if not?

👍 awesome! That's great news - I could do with some help from someone with better python skills to make this work on 2.7x and 3.x python. This branch 3.x doesn't work atm - due to string and byte changes in 3.x. I'll try and get it sorted out but if anyone is an expert and can make any suggestions, please do! :)

robertsosinski · 2019-06-22T00:24:56Z

Hi again,

So I installed from source with pip install -e .[u2f], is there anything else I should try?

Am I doing something else wrong? After it asks for the Yubikey, I just put it in and press the glowing Y symbol (where I see the input stream in) and it ends up asking again that no U2F key is found.

I've tried both Python 2.7.16 and 2.7.15.

Thanks for all your help!

-Robert

sinkr · 2019-06-22T16:25:31Z

@adcreare: [insert diety here] bless you!

I was finally able to get this to work by forcibly using Python 2.7!

mhenniges · 2019-06-25T12:43:04Z

@adcreare I did some hacking at this and with small changes I'm getting good results for both 2.7 and 3.7. I'm not sure what's the most convenient way to share those, but here's a diff:

--- a/aws_google_auth/google.py
+++ b/aws_google_auth/google.py
@@ -138,7 +138,10 @@ class Google:
         typeOfInput = type(input)
         if typeOfInput == dict:  # parse down a dict
             for item in input:
-                return Google.find_key_handle(input[item], challengeTxt)
+                rvalue = Google.find_key_handle(input[item], challengeTxt)
+                if rvalue is not None:
+                    return rvalue
+
         elif typeOfInput == list:  # looks like we've hit an array - iterate it
             array = list(filter(None, input))  # remove any None type objects from the array
             for item in array:
@@ -408,9 +411,10 @@ class Google:
 
         # txt sent for signing needs to be base64 url encode
         # we also have to remove any base64 padding because including including it will prevent google accepting the auth response
-        challenges_txt_encode_pad_removed = base64.urlsafe_b64encode(base64.b64decode(challenges_txt)).strip('=')
+        challenges_txt_encode_pad_removed = base64.urlsafe_b64encode(base64.b64decode(challenges_txt)).strip('='.encode())
+
         u2f_challenges = []
-        u2f_challenges.append({'version': 'U2F_V2', 'challenge': challenges_txt_encode_pad_removed, 'appId': appId, 'keyHandle': keyHandle})
+        u2f_challenges.append({'version': 'U2F_V2', 'challenge': challenges_txt_encode_pad_removed.decode(), 'appId': appId, 'keyHandle': keyHandle.decode()})
 
         # Prompt the user up to attempts_remaining times to insert their U2F device.
         attempts_remaining = 5

adcreare · 2019-06-27T11:56:42Z

@mhenniges that looks wonderful! I'll apply this diff tomorrow and update the PR.

adcreare · 2019-06-28T07:49:47Z

Updated to make python 3 work. Thanks again @mhenniges - ready for a review/merge

mhenniges · 2019-07-02T19:56:58Z

@stevemac007 - Anything else we can do to help here so this can merge in?

FabianFrank · 2019-07-03T00:00:16Z

I can confirm that with this PR merged login with a yubikey is working again.

jtatum · 2019-07-14T19:29:19Z

This resolved the error message for me, but on a system with no security key, auth now fails. I expected the --disable-u2f option to allow me to select some alternate auth method but it doesn't - I still get prompted to insert my key.

ahilsend · 2019-07-18T06:49:03Z

This fix solves the Yubikey auth for me.

lyonsy · 2019-07-19T12:06:03Z

This also fixed for me. Using this: https://www.yubico.com/product/yubikey-5-nfc

stevemac007

Looks good - can't personally verify, but community verification is good.

neilramsay · 2019-08-01T21:12:39Z

Release v0.0.32 works for my Yubikeys (firmware versions 4.3.5, and 5.1.1).
Thanks :-)

adcreare added 2 commits June 14, 2019 22:14

fixed yubikey auth flow

97b4b94

- added functions to handle custom field object - handle challange id value being single - various base64 decoding + base64 url encoding now required by google on post payload

removed unused code

956bff2

This was referenced Jun 14, 2019

id-challenge HTML element does not always contain a JSON value #128

Closed

Support for yubikey or google titan? #135

Closed

fixed lint items from build

1d0c98f

adcreare added 2 commits June 15, 2019 07:29

made python 3x compatible

368c44a

fixed linting again

c86ba41

remove padding if exists

1c34a19

- fixes random login error

Python 3.x compatible!

aa4ab76

ahilsend mentioned this pull request Jul 19, 2019

Fix yubikey tink-ab/aws-google-auth#1

Merged

stevemac007 approved these changes Jul 23, 2019

View reviewed changes

stevemac007 merged commit 4779703 into cevoaustralia:master Jul 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve issue 128 - yubikey auth flow broken by google page update #136

Resolve issue 128 - yubikey auth flow broken by google page update #136

adcreare commented Jun 14, 2019

coveralls commented Jun 14, 2019 •

edited

hadalin commented Jun 17, 2019

mhenniges commented Jun 17, 2019

adcreare commented Jun 17, 2019

mhenniges commented Jun 18, 2019

sinkr commented Jun 19, 2019 •

edited

adcreare commented Jun 19, 2019

sinkr commented Jun 19, 2019

mhenniges commented Jun 20, 2019 •

edited

adcreare commented Jun 20, 2019 •

edited

mhenniges commented Jun 21, 2019

robertsosinski commented Jun 21, 2019

robertsosinski commented Jun 21, 2019

adcreare commented Jun 21, 2019

adcreare commented Jun 21, 2019

robertsosinski commented Jun 22, 2019

sinkr commented Jun 22, 2019 •

edited

mhenniges commented Jun 25, 2019 •

edited

adcreare commented Jun 27, 2019

adcreare commented Jun 28, 2019

mhenniges commented Jul 2, 2019

FabianFrank commented Jul 3, 2019

jtatum commented Jul 14, 2019

ahilsend commented Jul 18, 2019

lyonsy commented Jul 19, 2019

stevemac007 left a comment

neilramsay commented Aug 1, 2019

Resolve issue 128 - yubikey auth flow broken by google page update #136

Resolve issue 128 - yubikey auth flow broken by google page update #136

Conversation

adcreare commented Jun 14, 2019

coveralls commented Jun 14, 2019 • edited

hadalin commented Jun 17, 2019

mhenniges commented Jun 17, 2019

adcreare commented Jun 17, 2019

mhenniges commented Jun 18, 2019

sinkr commented Jun 19, 2019 • edited

adcreare commented Jun 19, 2019

sinkr commented Jun 19, 2019

mhenniges commented Jun 20, 2019 • edited

adcreare commented Jun 20, 2019 • edited

mhenniges commented Jun 21, 2019

robertsosinski commented Jun 21, 2019

robertsosinski commented Jun 21, 2019

adcreare commented Jun 21, 2019

adcreare commented Jun 21, 2019

robertsosinski commented Jun 22, 2019

sinkr commented Jun 22, 2019 • edited

mhenniges commented Jun 25, 2019 • edited

adcreare commented Jun 27, 2019

adcreare commented Jun 28, 2019

mhenniges commented Jul 2, 2019

FabianFrank commented Jul 3, 2019

jtatum commented Jul 14, 2019

ahilsend commented Jul 18, 2019

lyonsy commented Jul 19, 2019

stevemac007 left a comment

Choose a reason for hiding this comment

neilramsay commented Aug 1, 2019

coveralls commented Jun 14, 2019 •

edited

sinkr commented Jun 19, 2019 •

edited

mhenniges commented Jun 20, 2019 •

edited

adcreare commented Jun 20, 2019 •

edited

sinkr commented Jun 22, 2019 •

edited

mhenniges commented Jun 25, 2019 •

edited