Falcon support

[ioquatix]

Hello. I'm the author of Falcon and I'm hoping to add support for DataDog's tracing.

1/ Have you tested DataDog with Falcon?
2/ Are you interested in helping with this?

[marcotc]

Hey @ioquatix, thank you for all your work on Ruby asynchronous I/O!

We have not tested ddtrace with Falcon yet, but we have interest in supporting it.
We haven't received any user feedback (until now) asking for Falcon instrumentation, or instrumentation for any other async I/O gem for that matter. But we are aware that of the progress in Ruby towards async I/O solutions, as we have already seen in other languages we support here at Datadog.

From a technical standpoint, implementing Falcon instrumentation only carries one caveat around ensuring ddtrace thread-local data structures stay consistent. But otherwise, instrumenting the interesting parts of Falcon (and/or its dependencies: nio4r, async-io, async, etc.) by creating a Span around the relevant code paths would likely suffice to provide visibility into the framework.

That being said, our team is not currently focused on working on this new instrumentation, but we are more than happy to assist with this work and help with any blockers necessary to get it merged.

[ioquatix]

Can you tell me how I could add tests to falcon? e.g. some high level integration tests which run some requests, log the spans to a local file or something, and then I can check they were captured correctly.

[marcotc]

👋 @ioquatix, had some dedicated time to try to write some specs you could follow, but I realized, after spending some time with the Async library, that our existing instrumentation dependent on thread-local variables would likely not work as expected with Falcon.

A Fiber#yield today would maintain the instrumentation context, meaning any code executed in other fibers would get assign as children of the caller of Fiber#yield. I can foresee this yielding a highly intertwined result.

The code responsible for the thread-local behaviour is encapsulated in this class:

dd-trace-rb/lib/ddtrace/context_provider.rb

Lines 41 to 48 in 0574335

	def local=(ctx)
	Thread.current[@key] = ctx
	end
	# Return the thread-local context.
	def local
	Thread.current[@key] ||= Datadog::Context.new
	end

The implementation of a Fiber-aware ContextProvider doesn't seem too complicated, the main challenge being to ensure no behaviour change for non-Fiber users.

I think this is the very first challenge we face to implement support for users of Async.

[ioquatix]

Just gonna drop the bomb. Did you know Thread.current#[] is actually fiber local?

[marcotc]

No, I did not! I thought it was only the other way around... thank you for that!

I'll try next to find what level we would like to instrument Falcon: I noticed while working with it yesterday that a lot of the heavy lifting is done by other "Socketry", and we might have better luck with those.

[ioquatix]

I have a plan to expose instrumentation hooks.

This is APM agnostic. I don't want APM libraries injecting random stuff into the async code if possible. I'd rather have a well defined interface for it.

I don't know what that is yet, but interested to hear how you get on.

[ioquatix]

Okay, I've created my agnostic hooks:

https://github.com/socketry/traces

I've implemented a backend for Datadog:

https://github.com/socketry/traces-backend-datadog

I've added some test trace captures including distributed tracing to https://github.com/socketry/async-http.

This would provide a decent coverage within Falcon, at least initially. But there are some wider issues including metrics, and this is all largely untested. Are you able to give me some credit on my plan so I can test it further?

[ioquatix]

By the way, with the advent of the above gems, this feature request is no longer relevant since I'll directly provide the tracing/metrics/integrations rather than expecting this gem to provide them.

[ivoanjo]

This would provide a decent coverage within Falcon, at least initially. But there are some wider issues including metrics, and this is all largely untested. Are you able to give me some credit on my plan so I can test it further?

Hey!

I was asking internally and there's actually a program at Datadog to give away free (as in beer/gratis) accounts for relevant open-source projects, which fits really well with Falcon and the Socketry ecosystem.

Do you mind filling https://www.datadoghq.com/partner/open-source/ and dropping a note in the comments that you've been working on this with me? We should be able to get you setup nicely :)

[ioquatix]

@ivoanjo I am still waiting, should I have heard something by now? Thanks!

[ivoanjo]

I shall find out what's up! :)

[ivoanjo]

Should be all set now! 🚀