Add a timeout to shutdown

[adamw]

Partially resolves #368

[lbialy]

A test would be nice. It doesn't have to actually deadlock, you could just sleep, capture IEx on user's main fiber and continue sleeping until some deadline (longer than the timeout provided in settings, eg timeout 100ms, user's fiber sleep deadline 200ms)

[adamw]

Hm I'm not sure if this would work, as when the timeout passes, we're not interrupting anything (well we're interrupting the waiting for the main thread to finish), but that main thread will continue working, and will prevent the scope from exiting. That's different from an exit process, which will just kill everything when the shutdown hook finishes.

[lbialy]

That's why I proposed that instead of a timeout we just race the interrupt with sleep(timeout)+Runtime.halt. The way you've implemented it can do the same if you call Runtime.halt after the warning to stderr is printed. The situation is that it is indeed a failure situation (in cleanup protocol) if that timeout is crossed so we can just kill the jvm there and then. We then avoid the problem of mainFiber.joinEither() completely.

[adamw]

Hm well I'm not convinced to do .halt on our own. If the shutdown hook exits, we just let the JVM do its job "normally". .halt is an intervention that signals we need to exit immediately, and we don't know that. So the less we change the "normal" flow, the better.

[adamw]

Ok, you can write a test, maybe not perfect, but we are testing some parts

[lbialy]

So once the shutdown hook finishes, the deadlock resolves and the main fiber joins correctly?

[adamw]

the main fiber, after interrupting, sleeps for 200ms, which is longer than the timeout (100ms). So we aren't simulating a deadlock, rather a cleanup that's longer

[lbialy]

Sure, I'm not talking about the test, the test is fine. I was thinking about the System.exit problem. Does timeout in the shutdown hook thread fix it?

[adamw]

Ah sorry ;) Yes of course, I tested that manually