OAuth to PCF UAA broke in 3.9.2

[gaigaslab]

We just upgraded from 3.8.0 to 3.9.2 using the bosh release.

We had previously setup oauth authentication with PCF UAA, but we are now getting:
"failed to exchange token"

I don't know where to look. I appear to be getting a token from UAA.

This is our configuration for each team:
fly -t app-main set-team -n ${myprojectname}
--uaa-auth-client-id concourse-app-1
--uaa-auth-client-secret
--uaa-auth-auth-url https://login.sys../oauth/authorize
--uaa-auth-token-url https://login.sys../oauth/token
--uaa-auth-cf-url https://api.sys.
--uaa-auth-cf-ca-cert ./Hosting_Services_CA.crt
--uaa-auth-cf-space ${myspaceguid}
--basic-auth-username
--basic-auth-password

Basic Auth continues to work.

[jama22]

Doesn't look like our pipelines cover UAA. When we investigate this, we should also consider writing tests to validate UAA compatibility

[pivotal-jwinters]

@jama-pivotal We just verified that this still works for us on 3.11.0. Not sure testing this in our pipeline is all that practical since we'd need a pcf instance to test against...

@gaigaslab Can you check the logs from the atc. You should see a message like "atc.skymarshal.oauth-callback.callback.failed-to-exchange-token". Can you check the error message it prints?

[jama22]

@pivotal-jwinters thanks for checking! I think you're right, it might not actually be practical to do it. Would still like to hear back from @gaigaslab about the details on their error. If there is a problem with 3.9.2 and Pivotal's UAA

[pivotal-jwinters]

@jama-pivotal there doesn't seem to be any significant changes from 3.9.2 to 3.11.0 for what its worth. It could be a certificate issue. If the certificate validation fails it gives the "failed to exchange token" error, but will output a proper message in the atc logs, so now we wait...

[gaigaslab]

@jama-pivotal @pivotal-jwinters

I mucked my way through accessing and finding the logs:

From:
/var/vcap/data/sys/log/atc/atc.stdout.log

{"timestamp":"1523996304.364424229","source":"atc","message":"atc.skymarshal.oauth-callback.callback.failed-to-exchange-token","log_level":2,"data":{"error":"oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"unauthorized","error_description":"Bad credentials"}","session":"10.4.44"}}

[pivotal-jwinters]

@gaigaslab can you try re-running the fly set-team command with your UAA credentials and see if that makes a difference?

Actually re-reading your initial message it looks like you tried this already?

[gaigaslab]

@pivotal-jwinters
Yes, I have tried that many times. I have tried recreating the team(s).

Is there anyone who can reproduce this issue in their environment? I have reproduced it in two of my environments.

[pivotal-jwinters]

@gaigaslab what version of UAA are you testing against? Are you able to get a token directly from UAA using the password grant?

[gaigaslab]

@pivotal-jwinters
From /debug/product_metadata
uaa v45.8 uaa-45.8.0-3468.25.0.tgz

As for getting 'a token directly from UAA using the password grant', I am not sure of the mechanics of this, but we can login via the uaac CLI and with 'cf login --sso' options. If you would provide a pointer to the mechanics for a curl-based method, I would be happy to test it out.

[pivotal-jwinters]

@gaigaslab are you able to get a token using uaac token owner get?

[gaigaslab]

@pivotal-jwinters
Yes - i.e. for the 'opsman' Client ID (which is not valid for App Manager or CF). I don't know how to use that command with SSO/SAML authentication, though.

[gaigaslab]

@pivotal-jwinters Accidentally hit the wrong button and closed this. It is not closed. It is still a bug!

[pivotal-jwinters]

@gaigaslab its no big deal, I was just trying to rule out a bad client id or client secret. Usually thats what {"error":"unauthorized","error_description":"Bad credentials"} means.

Do you have access to your UAA logs?

[gaigaslab]

Yes, I can get access to them.

…

On Fri, Apr 20, 2018 at 10:52 AM, Joshua Winters ***@***.***> wrote: @gaigaslab <https://github.com/gaigaslab> its no big deal, I was just trying to rule out a bad client id or client secret. Usually thats what {"error":"unauthorized","error_description":"Bad credentials"} means. Do you have access to your UAA logs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2097 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AfZUwNWnka9LBX9BL6FE0qPCvoE4EDLOks5tqfZBgaJpZM4SwHBn> .

[pivotal-jwinters]

@gaigaslab can you try logging in with concourse and see if there's anything interesting that shows up in the UAA logs?

[gaigaslab]

@pivotal-jwinters For some reason, I didn't get notified of this. I will try this in the next few hours.

[jhamon]

@pivotal-jwinters You should theoretically be able to test concourse against a standalone UAA bosh deployment. UAA has no dependency on the rest of PCF/CF.

[gaigaslab]

@jhamon @pivotal-jwinters
The problem was (apparent) incorrect handling of special characters in the UAA Client Secret. We worked around it by changing the password. Our old working password pre-3.9.2 was supplied to the concourse developers for diagnosis and root cause.

[pivotal-jwinters]

@jhamon the uaa provider is actually a poorly named cf provider since it does authz based on org/space membership. So unfortunately we need CF :(. We could probably get away with using cf dev, but its not a huge priority since we're revamping the way auth works anyway

[jccarte]

I ran into the same issue as @gaigaslab, removing special characters from my password is a workaround.

[akamalov]

@pivotal-jwinters, I am facing exactly the same issue, but having password without special characters is not solving the issue. I am running Concourse 3.14.0 against PCF 2.0. I get the following error:

"source":"atc","message":"atc.skymarshal.oauth-callback.callback.failed-to-exchange-token","log_level":2,"data":{"error":"oauth2: cannot fetch token: 401 Unauthorized\nResponse:{\"error\":\"unauthorized\",\"error_description\":\"Bad credentials\"}","session":"4.4.20"}}

My --uaa-auth-client-secret password1234 does not contain any special characters (I just set it to this to see if it works), but nope, it does not work. This is a huge issue for us. Technically, none of teams will be able to leverage Concourse to do their releases against PCF.