[FLINK-31656][runtime][security] Obtain delegation tokens early to support external file system usage in HA services #22298
Conversation
| @@ -213,6 +213,20 @@ public void obtainDelegationTokens(DelegationTokenContainer container) throws Ex | |||
| LOG.info("Delegation tokens obtained successfully"); | |||
| } | |||
|
|
|||
| @Override | |||
| public void obtainDelegationTokens() throws Exception { | |||
There was a problem hiding this comment.
We need to the immediately propagate the existing tokens (fetched here) to Task managers once start function is called. Otherwise, I see No credential error in task manager. I guess it is due to the latency of additional token fetching in startTokenUpdate function.
There was a problem hiding this comment.
At the one-time token obtain stage there are no task managers but I see that there is an issue so I'll take a look...
There was a problem hiding this comment.
I've just added further changes to solve the TM issue. We should test it on cluster in-depth because that's modifying Flink's critical path.
There was a problem hiding this comment.
@HuangZhenQiu please confirm back that you see working cluster tests on your side just to double check.
There was a problem hiding this comment.
@gaborgsomogyi Sorry for late reply. The feature is fully tested end to end.
gaborgsomogyi
force-pushed
the
FLINK-31656
branch
2 times, most recently
from
f3d83f1
to
9daabcd
Compare
gaborgsomogyi
changed the title
|
I've just fixed the unit tests + changed the title and description. The cluster tests are green. |
| // Obtaining delegation tokens and propagating them to the local JVM receivers in a | ||
| // one-time fashion is required because BlobServer may connect to external file | ||
| // systems | ||
| delegationTokenManager.obtainDelegationTokens(); |
There was a problem hiding this comment.
I don't seem to find any test for this new behaviour, would be good to add something to guard against accidental regressions in the future.
There was a problem hiding this comment.
Added that we obtain tokens in ClusterEntrypointTest but it would be overkill to check that token obtain happens before HA services.
…pport external file system usage in blob server
gaborgsomogyi
force-pushed
the
FLINK-31656
branch
from
9daabcd
to
6466d49
Compare
What is the purpose of the change
At the moment there are no delegation tokens available when HA services is starting. If the HA services uses an external file system where the authentication type is delegation token based (typically S3) then it throws and exception since there are no credentials.
In this PR I've moved the delegation token manager initialization before HA services and trigger a manual token obtain + local JVM receiver propagation. Additionally deferred base directory creation in
FileSystemBlobStoreandFileSystemJobResultStore.Brief change log
FileSystemBlobStoreandFileSystemJobResultStore. This is the solution for the task manager side.DelegationTokenManagerAPI documentation for better clarityVerifying this change
Manually on cluster.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): yesDocumentation