Accessing OIDC tokens from Pulsar

[SergeyYakubov]

Now that we have OIDC tokens available and refreshed in Galaxy (after PR #15300 is merged), we want to make them accessible in Pulsar so that we can use them to authenticate users and access resources on a remote machine.

The problem is that we cannot just send an ID token with job metadata since it might expire before it is needed. So, we've discussed this internally and see two options here:

1. Pulsar refreshes the token. For this, we'd need to send identity provider secrets to Pulsar. We'd have to do that for each job, similar to how Galaxy sends object store config with each job, because we cannot make it Pulsar settings since this does not scale and is not secure. Additionally, Pulsar would need to know how to refresh.
2. Create a new API endpoint in Galaxy to send a fresh ID token to Pulsar on request (see a diagram below). Galaxy would verify the job key similar to what happens when Pulsar sends files to Galaxy (https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/job_files.py#L117).

We want to implement option 2, but we would like to discuss it here first.

[hexylena]

second option makes a lot of sense to me

[mvdbeek]

Agreed, @jmchilton do you have any reservations here ?