live module cascading failures

[garlick]

If a broker slows down for a while then returns to normal responsiveness, the following may happen (test by sending SIGSTOP to a broker, then SIGCONT):

$ ./flux start -s64 
$ kill -STOP 37817
$ flux-start: 1 (pid 37817) Stopped (signal)

The broker's TBON parent may mark it 'failed', and its children may reparent. This is normal.

[1460150810.190432] live.crit[0]: transitioning 1 from OK to SLOW
[1460150814.190372] live.crit[0]: transitioning 1 from SLOW to FAIL
[1460150814.193420] broker.crit[3]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)
[1460150814.196075] broker.crit[4]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)

When the broker becomes responsive, it may decide its TBON children have failed. This is wrong.

$ kill -CONT 37817
flux-start: 1 (pid 37817) Continued
jimbo /home/garlick/proj/flux-core/src/cmd > [1460150826.819672] live.crit[1]: transitioning 3 from OK to SLOW
[1460150826.819703] live.crit[1]: transitioning 4 from OK to SLOW
[1460150826.820033] live.crit[1]: transitioning 4 from SLOW to FAIL
[1460150826.819672] live.crit[1]: transitioning 3 from OK to SLOW
[1460150826.820007] live.crit[1]: transitioning 3 from SLOW to FAIL

This triggers their children to reparent:

[1460150826.826241] broker.crit[8]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)
[1460150826.827488] broker.crit[9]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)
[1460150826.828682] broker.crit[10]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)
[1460150826.829373] broker.crit[7]: reparent ipc:///tmp/flux-37789-FbuCX8/0/req (new)
[1460150828.189251] live.crit[0]: transitioning 1 from FAIL to OK

Further, the second set of failed nodes remain that way.

$ flux up
ok:     [0-2,5-63]
slow:   
fail:   [3-4]
unknown:

[garlick]

I'm going to suggest that for the 0.3.0 release we avoid loading the live module, as its efforts to keep the session wired up minus failed nodes will be in vain without some higher level mechanism to shrink the job, and its design was built on having a separate event "bus" but we've disabled epgm by default and are distributing events via the TBON.

[grondo]

That seems fine, however it might be useful to have a fallback for flux up or have the command check for existence of the live module or in some other way indicate that the "up" service isn't functional to avoid confusion.

[garlick]

Yeah, we can assume that the entire session is up if there's nothing monitoring liveness to tell us otherwise, so that's easy.

[garlick]

Live module is no longer loaded by default. It should probably just be removed. Somebody please euthanize it.