CDK: SingleUseRefreshTokenOauth2Authenticator update config with access tokens and expiration date

[alafanechere]

What

Closes #20914

Before this PR the SingleUseRefreshTokenOauth2Authenticator eagerly performed access token refresh:

• In each operation (check, discover, read) a new OAuth flow is started even if the previous access token is still valid.

As synchronous operation, like check on source creation, can't process control message, we can end up with invalid refresh token as the latest one are not persisted to the DB. (more details in https://github.com/airbytehq/airbyte-internal-issues/issues/1260)
To mitigate this problem we try to store the expiration date of the access token in the config and only perform refresh when the expiration date is met.

How

• Read access_token_expiration_datetime from connector configuration
• On access token refresh:
  • Update the access_token_value
  • Update the access_token_expiration_datetime

Recommended reading order

airbyte/airbyte-cdk/python/airbyte_cdk/sources/streams/http/requests_native_auth/oauth.py

Line 93 in 833d978

class SingleUseRefreshTokenOauth2Authenticator(Oauth2Authenticator):

airbyte/airbyte-cdk/python/unit_tests/sources/streams/http/requests_native_auth/test_requests_native_auth.py

Line 183 in 833d978

class TestSingleUseRefreshTokenOauth2Authenticator:

🚨 User Impact 🚨

The connector using SingleUseRefreshTokenOauth2Authenticator should be reworked:
#7506

[pedroslopez]

Nice- main thing I think should be addressed is making sure that the initial access token is loaded from the config.

Should we always expect the expiration date to be part of the config or should we allow connectors to decide to use an an endpoint to retrieve the expiration date of the token?

Also, did you check the APIs for whether they had these kinds of endpoints like you mentioned? If so, I'm curious what the results of that investigation were.

[pedroslopez]

We should also set the initial access_token from the config as part of the initialization - otherwise I believe we wouldn't have an access token and require it to refresh, right?

GitLab was doing this manually:

airbyte/airbyte-integrations/connectors/source-gitlab/source_gitlab/source.py

Lines 57 to 59 in 5000dfb

	now = pendulum.now()
	self.access_token = access_token
	self.set_token_expiry_date(now.add(seconds=access_token_info["expires_in"]))

[alafanechere]

I made the change and added a test to ensure access_token attribute is set on init.

[pedroslopez]

note: this means as currently implemented during the sync the platform will perform 3 separate config updates (api calls) for each of these pieces

[alafanechere]

I discarded the use of ObservedDict and directly call a function from this method to emit the control message.

[alafanechere]

Should we always expect the expiration date to be part of the config or should we allow connectors to decide to use an an endpoint to retrieve the expiration date of the token?

Yes, we should expect the expiration date to be part of the config if the SingleUseRefreshTokenOauth2Authenticator is directly used.
I think that sources having a dedicated endpoint to fetch expiration can override the token_has_expired method.

Also, did you check the APIs for whether they had these kinds of endpoints like you mentioned? If so, I'm curious what the results of that investigation were.

I did but did not find these endpoints for Quickbook, TypeForm, Jira and Pipedrive

[evantahler]

This is good work! Do you happen to know how common it is for oAuth endpoints to return an expires_in / expires_at? If this is common, than storing this timestamp along with the token in the config is the way to go. Otherwise, perhaps we need to continue eagerly refreshing, but not in CHECK and DISCOVER modes (only READ and WRITE)

[alafanechere]

Do you happen to know how common it is for oAuth endpoints to return an expires_in / expires_at?

I'm pretty sure returning an expires_in field after an access token refresh is part of the OAuth standard.
I double-checked that it's the case for the new connectors we want to support and they do all return this field in their responses:

✅ Quickbook
✅ Jira
✅ Typeform
✅ Pipedrive

[alafanechere]

/publish-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/3828092811
✅ https://github.com/airbytehq/airbyte/actions/runs/3828092811

[alafanechere]

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3828134324
✅ https://github.com/airbytehq/airbyte/actions/runs/3828134324