[SPARK-28639][CORE][DOC] Configuration doc for Barrier Execution Mode

[sarutak]

What changes were proposed in this pull request?

SPARK-24817 and SPARK-24819 introduced new 3 non-internal properties for barrier-execution mode but they are not documented.
So I've added a section into configuration.md for barrier-mode execution.

How was this patch tested?

Built using jekyll and confirm the layout by browser.

[sarutak]

I put this section below the scheduling section because this is related to scheduling.

[sarutak]

CC: @jiangxb1987

[SparkQA]

Test build #108730 has finished for PR 25370 at commit b813d65.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

This is probably good, but are you sure they're meant to be public / documented?

[sarutak]

I think so because private properties are marked as internal like

  private[spark] val DRIVER_RESOURCES_FILE =
    ConfigBuilder("spark.driver.resourcesFile")
      .internal()
      .doc("Path to a file containing the resources allocated to the driver. " +
        "The file should be formatted as a JSON array of ResourceAllocation objects. " +
        "Only used internally in standalone mode.")
      .stringConf
      .createOptional

but those properties are not marked.

In particular, spark.scheduler.barrier.maxConcurrentTasksCheck.maxFailures should be public for users who want jobs fail fast.

But I'd like to confirm to @jiangxb1987 who introduced those properties, whether he intended they are public, just in case.

[jiangxb1987]

LGTM

[jiangxb1987]

Thanks for adding the document, I agree we shall allow end user to customize these config values, though I won't encourage changing the default value.

[dongjoon-hyun]

Shall we remove the extra > at Property Name>?
If you don't mind, please generate the doc and post the screenshot into the PR description.
That is helpful to spot this kind of typo-related bugs and the broken tables.

[sarutak]

Thanks. I've attached below.

[dongjoon-hyun]

<code>barrier()<code>?

[dongjoon-hyun]

<code>barrier()<code>.

[sarutak]

[SparkQA]

Test build #108865 has finished for PR 25370 at commit c987752.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kiszk]

nit: configed -> configured

[sarutak]

Thanks! Actually, this description is from here. So it might be better to modify it too.

[kiszk]

Yes, I think so. This is because other sections use configured.

[SparkQA]

Test build #108883 has finished for PR 25370 at commit 15d6909.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master