YARN Bug YARN-4301: NM disk health checker should have a timeout
The YARN disk health checker verifies a disk by executing mkdir and rmdir periodically.
If these operations does not return in a moderate timeout, the disk should be marked bad, and thus NodeHealthyStatus should flip to false.
We found this bug using Namazu, though the bug is not non-deterministic.
commit 96677be(hadoop-docker-nightly:20151027)
Unlike other experiments, we don't use nmz run in this experiment, because we are not so much interested in non-determinism.
$ go build -o mypolicy mypolicy.go
$ mkdir -m 777 /tmp/nmzfs-orig /tmp/nmzfs-mnt
$ sudo ./mypolicy inspectors fs -autopilot config.toml -mount-point /tmp/nmzfs-mnt -original-dir /tmp/nmzfs-orig
After started Namazu, you can start the YARN testbed. The YARN uses /nmzfs/nm-local-dir as the yarn.nodemanager.local-dirs property.
$ docker build -t yarn_testbed yarn_testbed
$ docker run -i -t --rm -p 8042:8042 -v /tmp/nmzfs-mnt:/nmzfs yarn_testbed
Open http://localhost:8042/node in a web browser, and make sure that NodeHealthyStatus is true, and LastNodeHealthTime is updated per 30 seconds.
Then let's inject the fault to YARN, by sending SIGUSR1 to the mypolicy process.
$ sudo killall -SIGUSR1 mypolicy
After sending SIGUSR1, Namazu injects very long sleep (10 minutes) to each of filesystem operations in /nmzfs/nm-local-dir.
Unfortunately, YARN cannot detect such a disk failure.
The value of NodeHealthyStatus keeps true, though LastNodeHealthTime gets stuck.
We consider this behavior as a bug.
$ docker rmi yarn_testbed
$ sudo fusermount -u /tmp/nmzfs-mnt