Restart apps that died due to errors

[eugeneia]

See the commit comments.

Also see App Restart for prior thoughts.

[eugeneia]

This is probably a no brainer. Lua only gives us pcall so we use that. On failure we tag the app with its error and time of death. Once tagged, the app is considered dead.

[lukego]

Great! Looks like a good implementation with good unit tests.

Could you rebase onto master? Github says there is a conflict to resolve.

[eugeneia]

Yeah will do, also this might need a little more work/thinking now or later on. Two thoughts:

1

Should apps that fail to configure/start/stop be restarted? I tend to think so, consider this case: Due to a NFV configuration change a 65th Intel app is created and its configuration will fail, but another one will be stopped.(E.g. because we don't have control over the order in which compile_config_actions will spit out actions, and there is some hardware limit in intel apps). As of now snabb crashes with a message "pool overflow", but the app could also be configured again 2s later and run just fine.

Of course we'd need to keep track which "dead apps" failed during configure and not try to stop() them. Also this is a little bit of a recursive situation as apps can "die" during restarting.

My point is, as of now we can run into exceptional situations that are caused by a combination of individual app semantics and configure semantics and I suspect there will be more of these cases to come. Should it be part of configure semantics to always perform stop before start and the apps responsibilty to never fail as long as the configure behaves correctly (we have yet to define a semantic interface fo start, stop, ...)? And/Or should core.app handle these cases?

One the one hand, handling arbitrary error cases in core.app is sort of a "wildcard", we don't really know what its gonna do to future apps. On the other hand, if well designed, that can also be a good approach as it "fixes/handles" errors whose source is not even yet created. In a way core.app would accept a declaration of what the user wants, and log (non-fatal) errors if it fails to achieve those goals, but always keep trying. The user would debug/develop his apps and app network by figuring out how core.app can achieve his declaration of goals. Snabb would only crash due to bugs in Snabb itself, not bugs in individual apps itself (e.g. never crash. Its job would be to try implement the declared network, not to succeed at it). Maybe there will be an app that fails to start/stop/pull/push every 10th time for perfectly good real-world reasons. A future core.app might handle that app very well, and even report something smart as "app X causes overall decrease of network uptime to 99.9% and packet loss of 0.023%.". Could be a network engineers wet-dream feature (I imagine), Maybe you would implement faulty apps on purpose to simulate a "chaos-monkey" and test your network design.

This is getting pretty meta, not sure if my thoughts came across. Bottom line I feel there is big potential in availability safeguards in core.app and as of this PR it might not do enough.

2

I really don't like how I tag app.name... ;)

[eugeneia]

Rebased on master. I also fixed up my gripe with tagging app.name, its done only in one place ops:start. The previous code was simply unnecessary.