New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OAuth to PCF UAA broke in 3.9.2 #2097

Open
gaigaslab opened this Issue Mar 19, 2018 · 21 comments

Comments

6 participants
@gaigaslab
Copy link

gaigaslab commented Mar 19, 2018

We just upgraded from 3.8.0 to 3.9.2 using the bosh release.

We had previously setup oauth authentication with PCF UAA, but we are now getting:
"failed to exchange token"

I don't know where to look. I appear to be getting a token from UAA.

This is our configuration for each team:
fly -t app-main set-team -n ${myprojectname}
--uaa-auth-client-id concourse-app-1
--uaa-auth-client-secret
--uaa-auth-auth-url https://login.sys../oauth/authorize
--uaa-auth-token-url https://login.sys../oauth/token
--uaa-auth-cf-url https://api.sys.
--uaa-auth-cf-ca-cert ./Hosting_Services_CA.crt
--uaa-auth-cf-space ${myspaceguid}
--basic-auth-username
--basic-auth-password

Basic Auth continues to work.

@jama22

This comment has been minimized.

Copy link
Member

jama22 commented Apr 16, 2018

Doesn't look like our pipelines cover UAA. When we investigate this, we should also consider writing tests to validate UAA compatibility

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 17, 2018

@jama-pivotal We just verified that this still works for us on 3.11.0. Not sure testing this in our pipeline is all that practical since we'd need a pcf instance to test against...

@gaigaslab Can you check the logs from the atc. You should see a message like "atc.skymarshal.oauth-callback.callback.failed-to-exchange-token". Can you check the error message it prints?

@jama22

This comment has been minimized.

Copy link
Member

jama22 commented Apr 17, 2018

@pivotal-jwinters thanks for checking! I think you're right, it might not actually be practical to do it. Would still like to hear back from @gaigaslab about the details on their error. If there is a problem with 3.9.2 and Pivotal's UAA

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 17, 2018

@jama-pivotal there doesn't seem to be any significant changes from 3.9.2 to 3.11.0 for what its worth. It could be a certificate issue. If the certificate validation fails it gives the "failed to exchange token" error, but will output a proper message in the atc logs, so now we wait...

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 17, 2018

@jama-pivotal @pivotal-jwinters

I mucked my way through accessing and finding the logs:

From:
/var/vcap/data/sys/log/atc/atc.stdout.log

{"timestamp":"1523996304.364424229","source":"atc","message":"atc.skymarshal.oauth-callback.callback.failed-to-exchange-token","log_level":2,"data":{"error":"oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized","error_description":"Bad credentials"}","session":"10.4.44"}}

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 18, 2018

@gaigaslab can you try re-running the fly set-team command with your UAA credentials and see if that makes a difference?

Actually re-reading your initial message it looks like you tried this already?

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 19, 2018

@pivotal-jwinters
Yes, I have tried that many times. I have tried recreating the team(s).

Is there anyone who can reproduce this issue in their environment? I have reproduced it in two of my environments.

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 19, 2018

@gaigaslab what version of UAA are you testing against? Are you able to get a token directly from UAA using the password grant?

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 19, 2018

@pivotal-jwinters
From /debug/product_metadata
uaa v45.8 uaa-45.8.0-3468.25.0.tgz

As for getting 'a token directly from UAA using the password grant', I am not sure of the mechanics of this, but we can login via the uaac CLI and with 'cf login --sso' options. If you would provide a pointer to the mechanics for a curl-based method, I would be happy to test it out.

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 19, 2018

@gaigaslab are you able to get a token using uaac token owner get?

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 20, 2018

@pivotal-jwinters
Yes - i.e. for the 'opsman' Client ID (which is not valid for App Manager or CF). I don't know how to use that command with SSO/SAML authentication, though.

@gaigaslab gaigaslab closed this Apr 20, 2018

@gaigaslab gaigaslab reopened this Apr 20, 2018

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 20, 2018

@pivotal-jwinters Accidentally hit the wrong button and closed this. It is not closed. It is still a bug!

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 20, 2018

@gaigaslab its no big deal, I was just trying to rule out a bad client id or client secret. Usually thats what {"error":"unauthorized","error_description":"Bad credentials"} means.

Do you have access to your UAA logs?

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 20, 2018

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 20, 2018

@gaigaslab can you try logging in with concourse and see if there's anything interesting that shows up in the UAA logs?

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 24, 2018

@pivotal-jwinters For some reason, I didn't get notified of this. I will try this in the next few hours.

@jhamon

This comment has been minimized.

Copy link

jhamon commented Apr 24, 2018

@pivotal-jwinters You should theoretically be able to test concourse against a standalone UAA bosh deployment. UAA has no dependency on the rest of PCF/CF.

@gaigaslab

This comment has been minimized.

Copy link
Author

gaigaslab commented Apr 24, 2018

@jhamon @pivotal-jwinters
The problem was (apparent) incorrect handling of special characters in the UAA Client Secret. We worked around it by changing the password. Our old working password pre-3.9.2 was supplied to the concourse developers for diagnosis and root cause.

@pivotal-jwinters

This comment has been minimized.

Copy link
Contributor

pivotal-jwinters commented Apr 25, 2018

@jhamon the uaa provider is actually a poorly named cf provider since it does authz based on org/space membership. So unfortunately we need CF :(. We could probably get away with using cf dev, but its not a huge priority since we're revamping the way auth works anyway

@jama22 jama22 added this to Icebox in Operations via automation May 2, 2018

@jccarte

This comment has been minimized.

Copy link

jccarte commented May 4, 2018

I ran into the same issue as @gaigaslab, removing special characters from my password is a workaround.

@akamalov

This comment has been minimized.

Copy link

akamalov commented Jun 23, 2018

@pivotal-jwinters, I am facing exactly the same issue, but having password without special characters is not solving the issue. I am running Concourse 3.14.0 against PCF 2.0. I get the following error:

"source":"atc","message":"atc.skymarshal.oauth-callback.callback.failed-to-exchange-token","log_level":2,"data":{"error":"oauth2: cannot fetch token: 401 Unauthorized\nResponse:{\"error\":\"unauthorized\",\"error_description\":\"Bad credentials\"}","session":"4.4.20"}}

My --uaa-auth-client-secret password1234 does not contain any special characters (I just set it to this to see if it works), but nope, it does not work. This is a huge issue for us. Technically, none of teams will be able to leverage Concourse to do their releases against PCF.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment