Add Kafka integration

[tjwp]

This change adds a kafka integration based on the ruby-kafka gem.

The implementation follows the template of the existing racecar integration which also builds on top of ActiveSupport::Notifications.

The motivation for this change was to provide more detailed information on interactions with Kafka. We use racecar but for publishing we have implemented our own "outbox pattern". This new integration provides more visibility into the message delivery to Kafka.

Support is included for all of the blocks that ruby-kafka instruments. Sorry about the size of the change :)

The events are broken into internal modules based on the ruby-kafka class that emits them. I was originally thinking that the events could be separately enabled based on these modules, but ultimately decided not to implement that.

Currently the span names are prefixed by kafka., e.g. kafka.producer.deliver_messages but I could be convinced to drop that first part of the name.

CC @dasch

[delner]

@tjwp This is awesome! Thanks for putting this together. We'll definitely setup some time to review this as this is something I'd love to add to our library.

Before I get a chance to look deeper in the code, can you explain how ActiveSupport::Notifications plays in? The racecar instrumentation used this because it was the easiest way to hook in and generate spans. But from what I can see, ruby-kafka only has this listed as a development dependency?

[tjwp]

@delner As with racecar this was the easiest way to hook in to generate spans because ruby-kafka already contains instrumentation that is built on ActiveSupport::Notifications: https://github.com/zendesk/ruby-kafka#instrumentation

It is not a hard dependency for ruby-kafka but must required to use any of the provided instrumentation.

[tjwp]

I noticed the changes here #1075 and I've updated this branch to use the new contrib spec helper.

[delner]

@tjwp Gotcha, so this integration will only work if both ruby-kafka and activesupport of sufficient version is present?

If this is the case, we might need to make sure that this requirement is reflected appropriately in the Integration such that it'll raise warnings if c.use :kafka is activated but activesupport is not present.

[tjwp]

@tjwp Gotcha, so this integration will only work if both ruby-kafka and activesupport of sufficient version is present?

If this is the case, we might need to make sure that this requirement is reflected appropriately in the Integration such that it'll raise warnings if c.use :kafka is activated but activesupport is not present.

Is there a pattern for this that you can point me to for this? The code in this PR handles it the same way as the existing Racecar integration: https://github.com/DataDog/dd-trace-rb/pull/1070/files#diff-4b58bb2b027c784de6ca989f77238eb9R20-R22

[tjwp]

@marcotc Pushed the updates you mentioned.

It looks like there are some Excon related test failures that would be addressed by: #1083

[marcotc]

Looks great @tjwp! Could you rebase with master, since #1086 was merged, which will pick up the latest excon release containing a fix for our CI issues?
We should be ready to merge your PR right after that!

[tjwp]

@marcotc thanks, rebased!

[marcotc]

Thank you very much @tjwp! 🎉

We'll let you know when a release goes out including your changes.

[tjwp]

@marcotc Any update on when this may be included in a release? Thanks!

[marcotc]

Hey @tjwp, we are planning on releasing soon, tentatively next week. I'll keep you posted.

Thank you once again for contribution!