[FLINK-8744][docs] Generate "Common Option" section

[zentol]

What is the purpose of the change

With this PR the "Common Options" section is now generated by the ConfigOptionDocsGenerator.

Brief change log

• add ´CommonOption` annotation
• annotate a few options with added annotation
  • I was very conservative as i do not want to discuss in this PR which options should be regarded as common
• allow passing in a path prefix for testing purposes; this allows testing against Options* classes residing in the test scope
• extend the generator to gather all annotated options and generated a HTML table
• replace common section with generated file
• extend completeness test to also cover common section
• remove advanced section, as subsumed by full reference

Verifying this change

• run ConfigOptionsDocGeneratorTest#testCommonOptions
• run ConfigOptionsDocsCompletenessTest#testCommonSectionCompleteness

[StephanEwen]

Looks pretty good.

Is there a way we can "sort" the common options? Something like

• host:port (for standalone setups)
• java memory
• default parallelism / slots
• fault tolerance
• HA
• security

[zentol]

they should be sorted alphabetically based on the key, but i now see that this isn't actually the case.

I'm not sure if I can easily add the kind of sorting you're asking for as that would require the generator to know about the semantics of an option ("oh, this is a port option, but that one is about memory!").

[StephanEwen]

One could add an int to the annotation, as "priority / position" and sort by that.
Not sure nice, but could be okay.

I think it was nice for users that the most common options (the ones you need first) were at the top of the list.

Out of curiosity, what happens to options like env.java.opts which are shell script only options, but very common?

[zentol]

For env.java.opts we have a ConfigOption in the CoreOptions class. They are documented like any other option (and technically they are also usable like any other option).

[zentol]

@StephanEwen The options are now sorted.

[StefanRRichter]

LGTM 👍

[zentol]

merging.