[WiP] Refactor agent launchers, weave to use Observable

[SpComb]

Follows #2704

Fixes #2134 remaining launchers also call Docker::Container#start! and raise errors
Fixes #1639 weaveexec ps for expose migration no longer breaks with WEAVE_DEBUG=1 (stdout vs stderr)
Fixes #1397 weave launcher calls either launch-router or attach-router
Hopefully fixes #1395 by allowing weave actors to recover from crashes

This is a major refactoring that is likely to also include some new bugs, and needs testing before release.

• Simplify error handling by allowing actors to recover from crashes
• Untangle the Kontena::NetworkAdapters::Weave into separate Kontena::Launchers::Weave and Kontena::Workers::WeaveWorker with their own Observer/Observable states
  • The Kontena::Launchers::Weave Observable replaces the network_adapter:start notification / wait_weave_running? sleep (Weave router is running)
  • The Kontena::NetworkAdapters::Weave Observable replaces the network:ready notification / wait_network_ready? sleep (IPAM is available for allocations)
• Replace all wait_until! calls with Kontena::Observer#observe
• Have Kontena::Workers::WeaveWorker observe Kontena::Launchers::Etcd instead of dns:add notification... but this still fails across etcd/weave restarts ❗

TODO

• Testing
• Figure out if the weave/IPAM error handling is robust enough without the sleeps for API readyness

Refactor the ObservableObserver pattern

Many of the actors now do observe => update => update_observable/reset_observable, which needs better support from the Observer/Observable implementation.

• Fix observes to be exclusive

  Observer should wait for any still running observe block to complete before calling it again.

  Meanwhile, it can also coalesce updates that come in faster than the observer is able to observe them.

• Retry failed observes

  Some part of the Observer update should rescue retryable errors provide some sane retry policy.

  The wip update pattern rescues errors by logging them and setting reset_observable, but this only works for errors caused by out-of-date observables, relying on observable updates to retry... not a valid assumption.

  Letting the actor crash and having the celluloid supervisor restart it does not provide a sane retry policy: there is no backoff, and the actor crashes seem to leak memory: Agent: Actor crash-and-restart loop will eat lot of resources #2231

• Optimize deep observer chains with multiple node_info_worker observables to avoid unnecessary updates

  If the weave_launcher observes the node_info_worker, and the etcd_launcher observes both node_info_worker and weave_launcher, the etcd_launcher observe block should only run once both the node_info_worker and the weave_launcher have updated.

  This behaves correctly during the initial startup, but on subsequent node_info_worker updates, the etcd_launcher observe will also run a second time after the weave_launcher also updates.

• Optimize launchers that now update on each node info update, not just once at startup

  The various launchers subscribed to network_adapter:start, which the Kontena::NetworkAdapters::Weave only published once (unless it crashed). Now the same launcher observe blocks run on every node_info update. This means extra Docker container inspect calls.

• Smarter error handling for reporting agent health?

  Future PR...

[kke]

@ not needed, there's an attr_reader.

[SpComb]

Note: I don't really expect this PR to get reviewed/merged in this form. This is more about experimenting with how I think the agent could get refactored... once I think I have something working, I'll try and split out smaller PRs.

That may cause some extra work though, because a lot of this work is about the inter-dependencies between actors...