[FLINK-16742][runtime][dist] Extend and use BashJavaUtils to start JM JVM process and pass JVM memory args

[xintongsong]

What is the purpose of the change

This PR is part of FLIP-116. It makes the start-up shell scripts to launch JM JVM process with the new memory configuration logics and utils.

Brief change log

• 2dca2f2: Minor code clean-ups.
• ee0a086: Make config.sh not assuming BashJavaUtis always return two lines of results. The assumption was true for calculating TM resource, but is not for calculating JM resource.
• 56b965a: Extend BashJavaUtils for JM memory calculations, and use it in start-up scripts for standalone session cluster.
• 2b3ea1a: Move BashJavaUtils to a separated module. This is for resolving resolving the dependency between BashJavaUtils and flink-container.
• 18f44da: Extend BashJavaUtils for JM memory calculations, and use it in start-up scripts for standalone job cluster.
• d20feb4: Remove common memory configuration / calculation codes, which are no longer used, from config.sh.

Verifying this change

• Extended BashJavaUtilsITCase

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not applicable)

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 788ae1e (Wed Apr 15 11:40:19 UTC 2020)

Warnings:

• 1 pom.xml files were touched: Check for build and licensing issues.
• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• 46b64a4 UNKNOWN
• 4d8ef75 Travis: SUCCESS Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[xintongsong]

cc @azagrebin

[azagrebin]

Thanks for opening the PR @xintongsong
It looks quite good to me. I left couple of comments to discuss.
In particular, I am not sure about the need to support various ways to parse options for different Flink processes.

[xintongsong]

Thanks for the review @azagrebin.
I've addressed your comments.

[azagrebin]

Thanks @xintongsong for addressing the comments.
The PR looks good to me. I left only minor comments and a question about the contents of the bash util jar.

[xintongsong]

@azagrebin Thanks for the review. Comments addressed.

[xintongsong]

@azagrebin
I've addressed your comments. Please take a look at your convenience.

[azagrebin]

Thanks @xintongsong, I think the last change simplified a lot and the PR looks good overall.
I left only one concern about the -D option specifics.

[xintongsong]

Thanks @azagrebin, comments addressed.

[azagrebin]

Thanks @xintongsong , LGTM
I left only a small refactoring suggestion

[xintongsong]

Thanks @azagrebin, I've addressed the last comment.

[xintongsong]

Rebased onto #11577.
A few changes during the rebasing:

• Pass dynamic properties to BashJavaUtils for JM of standalone job clusters. (JM of standalone session cluster does not support dynamic properties ATM.)
• Add test case for getting JM resource parameters from BashJavaUtils with dynamic properties.

[azagrebin]

Looks like, we have to rebase this part on master because of FLINK-17023. Btw, wc -l solution there for counting lines looks a bit simpler.

[azagrebin]

extractExecutionResults() {
    local output="$1"
    local expected_lines="$2"
    local EXECUTION_PREFIX="BASH_JAVA_UTILS_EXEC_RESULT:"
    local execution_results
    local num_lines

    execution_results=$(echo "${output}" | grep ${EXECUTION_PREFIX})
    num_lines=$(echo "${execution_results}" | wc -l)
    if [[ ${num_lines} -ne ${expected_lines} ]]; then
        echo "[ERROR] The execution result has unexpected number of lines, expected: ${expected_lines}, actual: ${num_lines}." 1>&2
        echo "[ERROR] An execution result line is expected to have the prefix '${EXECUTION_PREFIX}'" 1>&2
        echo "$output" 1>&2
        exit 1
    fi

    echo "${execution_results//${EXECUTION_PREFIX}/}"
}

[xintongsong]

@azagrebin Sorry for the late response, and thanks for providing the solution for the conflict resolving. I've rebased the PR.

[azagrebin]

@flinkbot run travis

[azagrebin]

@flinkbot run azure