threaded tasks in job manager: support auto-refresh of auth token

[VictorVerhaert]

During a large scale production we end up with blocks where jobs don't get started due to Failed to start job 'j-2510101449364f5e9a844d1bc59e2653': OpenEoApiError('[401] TokenInvalid: Authorization token has expired or is invalid. Please authenticate again.')

The blocks are rather short making me think it only impacts all the threads that have already been created.

I think it might have something to do with the auth bearer not being thread safe when refreshing.
Adding a token getting in the thread with a locked getter (thread safe) could solve this, but it is certainly not the only solution here

[soxofaan]

I don't think this is technically a thread-safety problem. It's just that the bearer token is expired by the time the thread tries to use it, and the thread doesn't have the means to refresh it (like the main thread).

possible solution, I think:

• make sure to pass the thread a reasonably recent bearer token at thread creation time
• give the thread extra means to refresh the token when outdated
• allow the threads to ask the main thread for a recent bearer token "just in time"

[HansVRP]

would we then need a kind of token manger which can check if the token is valid and if not refresh?

[soxofaan]

make sure to pass the thread a reasonably recent bearer token at thread creation time

this is probably the easiest quickfix here, but there still remains risk that the bearer token expires before usage

give the thread extra means to refresh the token when outdated

would not be that hard, but the OIDC provider might not like the refresh-spamming that this might cause. There might also be risk that the refresh from one thread invalidates the tokens from other threads (including the main thread).

allow the threads to ask the main thread for a recent bearer token "just in time"

technically the best solution I guess, but makes the threading story more complex

[soxofaan]

would we then need a kind of token manger which can check if the token is valid and if not refresh?

we already have kind of this functionality already in place:

openeo-python-client/openeo/rest/connection.py

Lines 685 to 697 in 01e06b8

	try:
	# Initial request attempt
	return _request()
	except OpenEoApiError as api_exc:
	if (
	api_exc.http_status_code in {HTTP_401_UNAUTHORIZED, HTTP_403_FORBIDDEN}
	and api_exc.code == "TokenInvalid"
	):
	# Auth token expired: can we refresh?
	if isinstance(self.auth, OidcBearerAuth) and self._oidc_auth_renewer:
	msg = f"OIDC access token expired ({api_exc.http_status_code} {api_exc.code})."
	try:
	self._authenticate_oidc(

but it's not available in the threads (because of missing oidc_auth_renewer)

[VictorVerhaert]

FYI EUGW has been advised to pin their version to 0.43 again until this is fixed, as it causes 60% of job to remain created

[soxofaan]

I'm working on a PR at #818