Avoid warning message about invalid refuse_seconds value in Mesos >=0.21...

[MartinWeindel]

Starting with version 0.21.0, Apache Mesos is very noisy if the filter parameter refuse_seconds is set to an invalid value like -1.
I have seen systems with millions of log lines like

W0420 18:00:48.773059 32352 hierarchical_allocator_process.hpp:589] Using the default value of 'refuse_seconds' to create the refused resources filter because the input value is negative

in the Mesos master INFO and WARNING log files.
Therefore the CoarseMesosSchedulerBackend should set the default value for refuse seconds (i.e. 5 seconds) directly.
This is no problem for the fine-grained MesosSchedulerBackend, as it uses the value 1 second for this parameter.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Sounds reasonable, since the value is reported to be invalid. The intent seemed to be to set this to "unset" or something. 5 seems to do something different as it sets it to a concrete value. Knowing nothing about this, is there maybe a closer equivalent value like 0? or is it really best to set this to a fixed value?

[MartinWeindel]

The value 5 seconds is the default value of Mesos, which is used if not
set or an invalid value is given. So at least with current versions of
Mesos nothing changes in the behavior.

The parameter refuse_seconds configures how long Mesos should wait
before it offers resources again after the framework (i.e. here the
Spark scheduler backend) has refused them.
If you set it to 0, this means that Mesos will immediately offer these
resources again with the next allocation (by default after 1 second).
This will cause slightly higher traffic between the scheduler backend
and the Mesos master.
Alternatively, this parameter could be made configurable by Spark, but I
am not sure if it is really worth the effort.
In coarse grained mode, resources are allocated at the start. Are there
any circumstances other than a lost executor, where refused resources
will be used?

Am 20.04.2015 um 19:03 schrieb Sean Owen:

Sounds reasonable, since the value is reported to be invalid. The
intent seemed to be to set this to "unset" or something. 5 seems to do
something different as it sets it to a concrete value. Knowing nothing
about this, is there maybe a closer equivalent value like 0? or is it
really best to set this to a fixed value?

—
Reply to this email directly or view it on GitHub
#5597 (comment).

[srowen]

I suspect this is OK, since this was set up eons ago by @mateiz and so the right thing to do has probably changed. Your explanation makes sense. CC @jongyoul just in case.

[ash211]

Should a bug should be filed with Mesos to log the error once rather than repeatedly and filling up disk?

[srowen]

@ash211 That would be nice too. It seems like the -1 Spark sends is considered an invalid value, so it sounds like that much should still change.

[tnachen]

It's logged each time we recover resources in Mesos since that's when we evaluate when the filter should be applied. And yes Mesos needs a positive value for refuse seconds on the filter.

I think refused resources could be used when later there more resources available on from the same slave since someone else's task is gone. Also we're adding capacbility into coarse grained scheduler to launched multiple executors, and in addition to that we're putting in dynamic allocation into coarse grained scheduler too.

So we will be using refuse resources more often, but 5 as the default sounds reasonable to me.