Allow connection profiles to use custom STS and OIDC endpoints #14972

dkocher · 2023-08-07T14:29:54Z

Resolves #13804.

s3/src/test/resources/S3-OIDC-Google-Testing.cyberduckprofile

core/src/main/java/ch/cyberduck/core/KeychainLoginService.java

defaults/src/main/resources/default.properties

s3/src/test/resources/oidcTestcontainer/keycloak/Dockerfile

s3/src/main/java/ch/cyberduck/core/sts/STSAssumeRoleTokenExpiredResponseInterceptor.java

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

chenkins · 2023-08-09T12:33:28Z

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

+                    .find(host.getProtocol().getOAuthAuthorizationUrl()), this, prompt).build(), host)
+                    .withRedirectUri(host.getProtocol().getOAuthRedirectUrl())
+                    .withFlowType(OAuth2AuthorizationService.FlowType.valueOf(host.getProtocol().getAuthorization())));
+            configuration.addInterceptorLast(sts = new STSAssumeRoleCredentialsRequestInterceptor(oauth, this, trust, key));


As sts comes last, does this mean we renew oauth even if STS token is still valid? Is this necessary? Comment on this. Do we have tests for all possible cases?

OAuth should be refreshed first using the standard OAuth2ErrorResponseInterceptor although it must be verified this actually works as it only handles 401. We are missing tests cases.

OAuth should be refreshed first

This seems not self-evident to me - can you reason why?

If OAuth is refreshed, will this force to renew STS as well even if it is still valid? If yes, is this intended? Which cases should be tested?

Implementation currently is intended as follows

Refresh of OAuth tokens when expired preemptively in OAuth2RequestInterceptor ¹

Refresh STS tokens using cached OAuth token preemptively in STSAssumeRoleCredentialsRequestInterceptor ²

On error response handle cases in STSAssumeRoleTokenExpiredResponseInterceptor as follows

On 401, 403 refresh OIDC Id token

On 400 reuse cached OIDC Id token

For both cases above refresh STS tokens

Footnotes

https://github.com/iterate-ch/cyberduck/pull/14972/files#diff-7ec37d50b17b31a07e5438bce7f0f3f4eba8a930b1c100e12e3ad84dda666ff8R106 ↩

https://github.com/iterate-ch/cyberduck/pull/14972/files#diff-c51064e4ed9ed159481473d881ec27aae4bd319978568b45480976f8f989b043R89 ↩

Updated handling of authentication failures in S3AuthenticationResponseInterceptor as of 8ea744a.

Handle 403 and 400

Refresh STS token with cached OIDC Id token when not expired

Otherwise refresh OIDC Id token first

On failure refresh STS token with refreshed OIDC Id token

Google OpenID Provider AWS API HTTP Status Code Error Code (in XML response body)

OpenID Id Token Expired S3 ListObjects ¹

OpenID Id Token Expired STS AssumeRoleWithWebIdentity 400 ExpiredTokenException ²

STS Tokens Expired S3 ListObjects 400 ExpiredToken ³

STS Tokens Invalid S3 ListObjects 400 InvalidToken [^4]

HTTP/1.1 400 Bad Request x-amzn-RequestId: b6baa352-afcb-4763-a32c-a67d7eb5020b Content-Type: text/xml Content-Length: 346 Date: Sun, 13 Aug 2023 10:53:10 GMT Connection: close <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/"> <Error> <Type>Sender</Type> <Code>ExpiredTokenException</Code> <Message>Token expired: current date/time 1691923690 must be before the expiration date/time1691878565</Message> </Error> <RequestId>b6baa352-afcb-4763-a32c-a67d7eb5020b</RequestId> </ErrorResponse>

Footnotes

↩

Token expired: current date/time 1691790439 must be before the expiration date/time1691788167 ↩

The provided token has expired. ↩

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

s3/src/main/java/ch/cyberduck/core/s3/S3TokenExpiredResponseInterceptor.java

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

chenkins · 2023-08-10T15:47:13Z

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

+                    .find(host.getProtocol().getOAuthAuthorizationUrl()), this, prompt).build(), host)
+                    .withRedirectUri(host.getProtocol().getOAuthRedirectUrl())
+                    .withFlowType(OAuth2AuthorizationService.FlowType.valueOf(host.getProtocol().getAuthorization())));
+            configuration.addInterceptorLast(sts = new STSAssumeRoleCredentialsRequestInterceptor(oauth, this, trust, key));


OAuth should be refreshed first

This seems not self-evident to me - can you reason why?

If OAuth is refreshed, will this force to renew STS as well even if it is still valid? If yes, is this intended? Which cases should be tested?

s3/src/main/java/ch/cyberduck/core/s3/S3ExceptionMappingService.java

s3/src/main/java/ch/cyberduck/core/s3/S3Session.java

s3/src/main/java/ch/cyberduck/core/sts/STSAssumeRoleAuthorizationService.java

s3/src/test/resources/S3-OIDC-Testing.cyberduckprofile

s3/src/test/resources/testcontainer/docker-compose.yml

s3/src/main/java/ch/cyberduck/core/sts/STSAssumeRoleAuthorizationService.java

…be identified in keychain.

ylangisc

https://github.com/iterate-ch/cyberduck/pull/14972/files#r1299842670

…st class not working as expected). Signed-off-by: chenkins <chenkins44@gmail.com>

…ithWebIdentity testcontainer tests. Signed-off-by: chenkins <chenkins44@gmail.com>

dkocher requested a review from a team as a code owner August 7, 2023 14:29

dkocher marked this pull request as draft August 7, 2023 14:30

dkocher requested review from ylangisc and chenkins August 7, 2023 14:37

dkocher added this to the 8.7.0 milestone Aug 7, 2023

dkocher mentioned this pull request Aug 7, 2023

Sample connection profiles for S3 and OpenID Connect Federation iterate-ch/profiles#55

Closed

15 tasks

dkocher force-pushed the feature/GH-13804-oidc branch 2 times, most recently from 6fb5dee to 450138b Compare August 8, 2023 20:50

chenkins reviewed Aug 9, 2023

View reviewed changes

dkocher force-pushed the feature/GH-13804-oidc branch 6 times, most recently from adff9aa to de8f6e1 Compare August 9, 2023 21:23

chenkins reviewed Aug 11, 2023

View reviewed changes

dkocher commented Aug 11, 2023

View reviewed changes

s3/src/main/java/ch/cyberduck/core/sts/STSAssumeRoleAuthorizationService.java Outdated Show resolved Hide resolved

dkocher commented Aug 11, 2023

View reviewed changes

s3/src/main/java/ch/cyberduck/core/sts/STSAssumeRoleAuthorizationService.java Outdated Show resolved Hide resolved

dkocher force-pushed the feature/GH-13804-oidc branch from 631eab4 to 204949f Compare August 11, 2023 10:15

chenkins mentioned this pull request Aug 11, 2023

Connect to Minio with temporary credentials from STS obtained with OpenID Connect token from Keycloak #14893

Merged

5 tasks

dkocher force-pushed the feature/GH-13804-oidc branch from dbc86fd to a4ad431 Compare August 11, 2023 11:57

dkocher marked this pull request as ready for review August 11, 2023 12:15

dkocher force-pushed the feature/GH-13804-oidc branch 3 times, most recently from 4297378 to bc2b5d2 Compare August 11, 2023 21:02

dkocher mentioned this pull request Aug 11, 2023

Authentication using AWS AssumeRole and GetSessionToken with AWS STS #8880

Closed

dkocher marked this pull request as draft August 12, 2023 08:03

dkocher force-pushed the feature/GH-13804-oidc branch from bc2b5d2 to 19c8fc9 Compare August 12, 2023 20:13

dkocher mentioned this pull request Aug 12, 2023

Sample connection profiles for S3 and OpenID Connect Federation iterate-ch/profiles#58

Merged

dkocher marked this pull request as ready for review August 15, 2023 08:38

dkocher and others added 14 commits August 28, 2023 09:38

Set subject from JWT token as username to make sure OAuth tokens can …

293741e

…be identified in keychain.

Inject OAuth TTL into TestContainer tests.

11b79b0

Implement equals.

1479ace

Review tests.

6516800

Temporary credentials are not saved.

eb35e00

Add abstraction for authentication strategy.

e2baf39

Review.

c157b42

Review.

81a8f3b

Fix test.

25a7576

Merge implementations retrieving latest credentials on error response.

a2eec98

Fix test.

3f7f77d

Logging.

1913156

Review error mapping to handle error codes.

03d9bca

Add tests.

44f7d7f

dkocher force-pushed the feature/GH-13804-oidc branch from 2ca4ef4 to ad1c2e5 Compare August 28, 2023 07:40

dkocher enabled auto-merge August 28, 2023 07:42

ylangisc requested changes Aug 28, 2023

View reviewed changes

dkocher and others added 7 commits August 28, 2023 09:50

Refactor flows.

5ccac05

Delete long running tests.

2bfb964

Bugfix STS testcontainer tests (BeforeClass annotation in abstract te…

72f2829

…st class not working as expected). Signed-off-by: chenkins <chenkins44@gmail.com>

Review test setup.

595c12e

Formatting.

7e572ca

Review hash code.

8150cf5

Remove obsolete keycloak.json manipulation before running AssumeRoleW…

bdfdcf0

…ithWebIdentity testcontainer tests. Signed-off-by: chenkins <chenkins44@gmail.com>

dkocher force-pushed the feature/GH-13804-oidc branch from ad1c2e5 to bdfdcf0 Compare August 28, 2023 07:50

Merge token ivar.

a999e86

dkocher requested a review from ylangisc August 28, 2023 10:09

ylangisc approved these changes Aug 28, 2023

View reviewed changes

dkocher merged commit f5df84d into master Aug 28, 2023
4 checks passed

dkocher deleted the feature/GH-13804-oidc branch August 28, 2023 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow connection profiles to use custom STS and OIDC endpoints #14972

Allow connection profiles to use custom STS and OIDC endpoints #14972

dkocher commented Aug 7, 2023

chenkins Aug 9, 2023

dkocher Aug 10, 2023

chenkins Aug 10, 2023

dkocher Aug 11, 2023

dkocher Aug 13, 2023

chenkins Aug 10, 2023

ylangisc left a comment

Google OpenID Provider	AWS API	HTTP Status Code	Error Code (in XML response body)
OpenID Id Token Expired	S3 ListObjects		¹
OpenID Id Token Expired	STS AssumeRoleWithWebIdentity	400	`ExpiredTokenException` ²
STS Tokens Expired	S3 ListObjects	400	`ExpiredToken` ³
STS Tokens Invalid	S3 ListObjects	400	`InvalidToken` [^4]

Allow connection profiles to use custom STS and OIDC endpoints #14972

Allow connection profiles to use custom STS and OIDC endpoints #14972

Conversation

dkocher commented Aug 7, 2023

chenkins Aug 9, 2023

Choose a reason for hiding this comment

dkocher Aug 10, 2023

Choose a reason for hiding this comment

chenkins Aug 10, 2023

Choose a reason for hiding this comment

dkocher Aug 11, 2023

Choose a reason for hiding this comment

Footnotes

dkocher Aug 13, 2023

Choose a reason for hiding this comment

Footnotes

chenkins Aug 10, 2023

Choose a reason for hiding this comment

ylangisc left a comment

Choose a reason for hiding this comment