Add Datadog::Worker and extensions

[delner]

Extracted from #879

This pull request adds the Datadog::Worker base class and its extension modules. It is intended to act as the foundation for background processes that run within the tracing library. Examples of these would be "writing traces to the agent" and "flushing runtime metrics to Statsd".

At its core, there is the Datadog::Worker class which itself defines a simple task, and a perform function.

Each background process would extend this base class, and define its work behavior within. e.g.

class TraceWriter < Datadog::Worker
  def perform(traces)
    # Write traces to agent
  end
end

Because each worker might need to behave differently (e.g. polling, queuing, threading etc) these behaviors are provided through modules Polling, Async::Thread, IntervalLoop, Queue which can then be composed into the worker. Each of these layers wrap the perform function with additional behavior, and provide new functions to the worker. e.g.

class AsyncTraceWriter < TraceWriter
  include Workers::Queue
  include Workers::Polling
end

class RuntimeMetrics < Datadog::Worker
  include Workers::Polling

  def perform
    # Flush runtime metrics
  end
end

As described in #879, some of the benefits of implementing this include:

• Allows us to eliminate the SyncWriter by having one common definition for trace flushing behavior, then having AsyncTraceWriter decorated upon that.
• AsyncTraceWriter can optionally automatically switch to synchronous writing when forked (addresses the need for SyncWriter in integrations like Resque.)
• Breaks the worker behaviors into smaller, composable behaviors (e.g. Async, Queue, IntervalLoop, etc) which should be easier to test and re-use, which will...
• ...allow us to extract and introduce a runtime metrics worker. These changes make building & maintaining the runtime metrics worker easier because it shares many of the same basic worker components

The Datadog::Worker and its other components will be used in another set of PRs which will introduce the TraceWriter and RuntimeMetrics workers.

[marcotc]

Great stuff, very well flushed out implementation. 👍 🎉

[marcotc]

Is it ok to use a simple array here, as it is not thread-safe?

[delner]

I suppose it depends on your worker implementation, and how (also if) it uses threads. The buffer itself should be made thread safe if you require it; we do this by overriding the buffer for the TraceWriter. This is just a default implementation.

[marcotc]

Awesome quality of life improvement with this. 💯

[marcotc]

Looking forward to seeing these changes in the wild! ⛵️🌊