Don't Mark run as `lost` if a key event is rejected

[josephjclark]

This is related to #2943. Not a fix, but a better and cheaper intermediate solution.

Sometimes when something goes wrong in a run, key events like step:start and step:complete (I think literally just those two? Or anything but a log?) will fail.

As soon a step:complete fails, right now, the run will become Lost, because all subsequent events will be rejected by lighting.

If we forget the complexity of idempotency, of retries, of strictness, why a particular event was rejected: the wider story is that as soon as a step:complete is rejected, Lightning must mark the run crashed (or killed?) rather than lost.

Because Lost means "we don't know what happened to this run". But that's inaccurate - we know exactly what happened to the Run - Lightning rejected its events.

[josephjclark]

@theroinaochieng @taylordowns2000 I'm going to recommend this as a fairly high priority fix on the Lightning side. Maybe priority for next sprint?

I'm not sure HOW we'll Kill the run from the lightning side. I don't know what messaging we need to show the user ( the run may have finished just fine, it's just the reporting that failed). I don't think we need to send a Kill event to the worker or anything.

[brandonjackson]

@josephjclark Thanks for this. I'd love a bit more information to assess the frequency/impact of this issue.

What % of outstanding lost runs do you think can be attributed to this?

[josephjclark]

Just speaking with Taylor and I think we ought to escalate this if we have the cycles @theroinaochieng

@brandonjackson my gut feeling is that this is causing like 90% of lost runs.

Fixing this doesn't fix any of the underlying errors - it just means we report them a bit more honestly.