[FLINK-30370][runtime][security] Move existing delegation token framework authentication to providers

[gaborgsomogyi]

What is the purpose of the change

In order to make a generic delegation token manager all authentication specific things must be extracted. This is such a step. In this PR I've moved Kerberos authentication to the provider side.

Brief change log

Moved Kerberos authentication to the provider side.

Verifying this change

Existing unit/integration tests + manually on minikube.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[gaborgsomogyi]

cc @mbalassi

[flinkbot]

CI report:

• 5693bf5 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[mbalassi]

@gaborgsomogyi with the externalization of the HBase connector will the HBaseDelegationTokenProvider continue to live in flink-runtime or should we move it to the hbase connector repo instead longer term (after 1.17)?

cc @ferenc-csaky

[gaborgsomogyi]

@mbalassi I've mentioned the move possibility to @ferenc-csaky but seems like not yet done. From my point of view we could move it now :)

[mbalassi]

@gaborgsomogyi please take a look at the CI:

Dec 13 08:04:01 [ERROR] Failures: 
Dec 13 08:04:01 [ERROR]   KerberosDelegationTokenManagerITCase.startTokensUpdateShouldScheduleRenewal:184 expected: <true> but was: <false>

[gaborgsomogyi]

Just fixed it.

[gaborgsomogyi]

Here the expectation is still 3 calls:

• Manual call in line 164
• Trigger again by exception (retry)
• Trigger again by normal re-schedule

[ferenc-csaky]

@gaborgsomogyi with the externalization of the HBase connector will the HBaseDelegationTokenProvider continue to live in flink-runtime or should we move it to the hbase connector repo instead longer term (after 1.17)?

cc @ferenc-csaky

I think this should not be involved in the first round of the externalization. We try to have the first externalized connector release being the same we had in 1.16, so the connector code can be removed in 1.17 already. I will just start to port the e2e tests, which has to be done before the connector release, so there are some work to be done on that front.

If there is capacity, PRs can be opened until then, I will be able to help after dealing with the e2e tests.

[gaborgsomogyi]

@ferenc-csaky ok, then we can come back to this later in 1.17. At that timepoint we're going to have a relatively stable API.

[mbalassi]

LGTM