[FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode

[RocMarshal]

What is the purpose of the change

Fix the invalid configuration when using initialized root logger level on yarn deployment mode

Brief change log

• Fill the containerized env values of root logger level before submiting yarn application.

Verifying this change

This change added tests and can be verified as follows:

• *Manually verified the change by running a 4 nodes cluster.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• a746744 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[1996fanrui]

Thanks @RocMarshal for the fix!

Flink does not support modifying the logger level only in yarn mode, right?

• If yes, the solution may be right.
• If flink on k8s doesn't support modifying the logger level as well. Current solution doesn't make sense to me.

[X-czh]

Hi @1996fanrui, I've verified that the current release of Flink supports modifying the logger level in both Native K8s and Standalone mode. Since YARN prepares the container launching context in a special way, we'll need to do some special adaptation anyway. LGTM

[1996fanrui]

Thanks @RocMarshal for the fix and @X-czh for the double-check!

@RocMarshal , could you rebase the master branch?

LGTM assuming the CI is green.

[RocMarshal]

Thanks @X-czh for the double-check and @1996fanrui review.

I made a fix like the env-value passing-in at RocMarshal@624c199

Would we merge the additional fix into the current pr ?
Please let me know what's your opinion~ :)

[1996fanrui]

I made a fix like the env-value passing-in at RocMarshal@624c199
Would we merge the additional fix into the current pr ?
Please let me know what's your opinion~ :)

Same PR is fine for me, but separate JIRA and commits are necessary.

Separate JIRA and commits are easy to follow up and cherry pick for other users.

[1996fanrui]

As I understand, this validator wanna check whether the value of configOption in config is legal.

• If it's legal, we can add the env
• If it's illegal, we will ignore.

Is my understanding right?

If right, I don't think we should do this check here. I mean we should cover Flink on Kubernetes if we wanna check whether it's legal. So check them in the yarn side doesn't make sense to me.

[1996fanrui]

In general, we check the option value only once. And all callers or users don't need to care about whether it's legal.

[RocMarshal]

Hi, @1996fanrui Thanks for your quick-review~

Is my understanding right?

Yes.

I have to admit that I didn't consider compatibility with on k8s from the beginning.
This is a trade-off between robustness and redundancy check for me.
To be honest, I would prefer to remove this validation here. 😭

[1996fanrui]

Removing is fine for me, the code will get clearer here.

[1996fanrui]

LGTM assuming the CI is green.