[FLINK-12139][Mesos] Add disk space parameter.

[juangentile]

What is the purpose of the change

Add a parameter to set the required disk space for a Mesos deployment. Before this change the default value is 0 with no option to change it. We maintain the default 0, we only make it parameterizable

Brief change log

• The new parameter is loaded in the class MesosTaskManagerParameters.java
• The new parameter is used in the class MesosResourceManager

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): don't know
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[tillrohrmann]

Thanks for this fix @juangentile. My only question is what the default value of Mesos is for the disk value if it is not specified. Before we did not specify it so we should make sure that this is the default value of MESOS_RM_TASKS_DISK_MB.

Did you test your changes on a Mesos cluster (w/ and w/o setting the disk size)?

[tillrohrmann]

What is the default Mesos value if disk is not specified?

[juangentile]

Hello Till,

Sorry for the delay in the response.
I tested flink on the mesos cluster we have at Criteo and before my change it didn't work because it was taking the default 0 value for the disk limit.
In our cluster we enforce the disk usage check but setting this parameter --enforce_container_disk_quota.
You can find the description of the parameter here: http://mesos.apache.org/documentation/latest/configuration/agent/
(and here http://mesos.apache.org/documentation/latest/configuration/agent/ - http://mesos.apache.org/documentation/latest/isolators/disk-du/)
The default installation of mesos has false for enforce_container_disk_quota so the disk limit is never checked.

After my change, it works correctly on our mesos cluster (if we set a value different from the default 0 value using the new parameter).

[tillrohrmann]

Hmm, looking at the Mesos source code it seems that the disk size will be automatically calculated if not specified. Thus, I would suggest to only set the "disk" resource value if the value is larger than 0.0.

[juangentile]

sure, I'll do the change :)

[tillrohrmann]

Thanks for commenting on my comment @juangentile. I think to be on the safe side we could only set the disk value if the value is different than 0.0.

[tillrohrmann]

Hmm, looking at the Mesos source code it seems that the disk size will be automatically calculated if not specified. Thus, I would suggest to only set the "disk" resource value if the value is larger than 0.0.

[tillrohrmann]

Thanks for addressing my comments @juangentile. LGTM. Merging this PR.