HeartBeat vs. mapreduce.task.timeout doesn't consider "0 == infinite" case

[rusnyder]

For the Hadoop config value "mapreduce.task.timeout" (or "mapred.task.timeout" for Hadoop 1), a value of 0 is the equivalent of disabling that timeout. However, the ES-Hadoop HeartBeat compares "mapred.task.timeout" to its own delay and throws an error if the heartbeat is greater than the task timeout.

I'm seeing this with 2.1.0.Beta3

org.elasticsearch.hadoop.mr.HeartBeat

class HeartBeat {
    ...
    HeartBeat(final Progressable progressable, Configuration cfg, TimeValue delay, final Log log) {
        Assert.notNull(progressable, "a valid progressable is required to report status to Hadoop");
        TimeValue tv = HadoopCfgUtils.getTaskTimeout(cfg);
        Assert.isTrue(tv.getSeconds() > delay.getSeconds(), "Hadoop timeout is shorter than the heartbeat");
        ...
    }
    ...
}

The assert should probably be or'ing w/ a "tv.getSeconds == 0" or something like that.

[costin]

Hi,

Thanks for the report. I've fixed this in master and on the 2.x branch and published the artifacts (available in maven). Can you please check it out?
Besides handling the infinite use case, I've also fixed an issue with the scheduling as the heartbeat was only activating one (per task) but now it should run continuously.

Cheers,

[rusnyder]

Nice! I just ran into that scheduling issue last night and hadn't yet diagnosed it, but looking at your commits now realize that you've fixed both of my problems in one swoop!

Thanks for the quick turnaround!

[costin]

Glad to help! Please let me know if the latest build solves your issue(s).

On 4/19/15 4:39 PM, rusnyder wrote:

Nice! I just ran into that scheduling issue last night and hadn't yet diagnosed it, but looking at your commits now
realize that you've fixed both of my problems in one swoop!

Thanks for the quick turnaround!

—
Reply to this email directly or view it on GitHub
#426 (comment).

Costin

[rusnyder]

Solved - things are running smoothly now.