Solve the startup and configuration hell.

[ivantopo]

There are a few inconveniences related to how Kamon's lifetime is handled (or poorly handled, to be more accurate):

• First, Kamon has to rely on some global state (currently inside the Kamon companion object) so that all instrumentation can easily use the APIs without having to make a big deal trying to find the right Kamon instance.
• Since there is a chance of instrumentation using some of Kamon's APIs before Kamon is effectively started, we broke the initialization in 2 steps: first is starting and configuring the tracing and metric modules, but not start the reporting modules until Kamon.start() is called. This is good in the sense that whenever you access any of Kamon's APIs you will be guaranteed that Kamon is started, but it took away the possibility of calling Kamon.start(config) which many users seem to be missing now. As a side note, we introduced config providers so that you can still kick in and provide a config object, but it might not be the most comfortable API.
• When Kamon is shutdown, it isn't shutdown very cleanly. The only thing it does is shutdown Kamon's ActorSystem, which is enough for stop reporting to the backends, but then Kamon cannot be started again. This is specially noticeable if you are running a Play! application with Kamon on Development, after the first reload Kamon will stop working. This is a very unacceptable user experience in my opinion.
• It shouldn't be required to have Kamon started for running tests. We used to force users to start Kamon to make sure that they do it only once and in the right place in the app, but when running tests it might get really tricky and in most cases it is actually unnecessary to have Kamon started.

Proposals (WIP):

Kamon should get a proper and deterministic startup and shutdown behavior. It should be very clear what to expect if you don't start Kamon, if you start it, or if you start it a hundred times, same goes with shutdown.

The shutdown process is the simplest to approach. When Kamon is shutdown, besides shutting down the internal ActorSystem we should also make sure that Kamon's internal state is put back to as if it was never started. We might consider extending the shutdown process to signal Kamon's shutdown to all reporting modules so that they can close any related resources. (Note that this could also be achieved by hooking in the reporting module's actor lifecycle). If we shutdown and start Kamon again it should work exactly as if it was started for the first time, except that all entities that were created with the first Kamon instance will not be linked to the new instance.

We should allow users to pass a custom config object. This is also required for Play! applications where we should always be using the config object form the context on the application loader. So, this leaves a few questions open:

• what happens if Kamon APIs are used before Kamon is started? My current idea is that Kamon could be in a "permissive mode" when it is not started, meaning that it could configure the metrics module with the default configuration and accept all entities for monitoring. Then, when Kamon is started, proper filters will be taken from the provided configuration object and they will be applied before dispatching subscriptions. This means that if the gap between using the APIs and starting Kamon is big, we could be tracking a lot of entities that actually don't need to be tracked, but in reality most applications will be initializing correctly and in the right order and this permissive mode will only be useful for tests.
• do we need to do anything special when running tests? I would say no, that permissive mode should be enough. No need to start anything and all will work.

Hope to get some feedback before jumping into implementing this, which I want to start as soon as possible.

[briantopping]

Is this the container for comments on configuration?

One of the issues I just found with using kamon.config-provider is the configs may be unresolvable without help from the application. Ironically, kamon.config-provider can help by providing the method that fills in the remaining values and calls Config.resolve, but if Kamon can't find the provider implementation because the configs can't be resolved... It's a circular dependency.

One solution may be to load the configs unresolved and use a manual means to find the kamon.config-provider value.

[ivantopo]

I think that there are certain things that are kind of defined already for this issue:

• config providers are not a very good idea, they should go away.
• having Kamon.start(config) is awesome, everybody likes it and we should keep it.

The only thing that we really need to answer is this question from above: "what happens if Kamon APIs are used before Kamon is started?"... the options are either to block everything or to allow everything until Kamon is properly started and all filters and settings are known.

I am very enthusiastic of going for the allow everything approach. And by "allow everything" I mean that if you request Kamon for an entity, it will give you the entity, if you start a trace, it will measure it but with metrics only mode. Subscriptions will not be working, not reporters will be started until you call Kamon.start(...). This also means that we should give a lot of love to the kamon-testkit module to easily get metrics out of there, since currently all of our tests are working using the subscriptions to gather the data.

[ivantopo]

same as #421, won't fix on 0.6 and needs to get #502 right on 1.0