Tail Workers implicitly activate node:diagnostics_channel publishers and add unrequested overhead

[astanciu]

What happened?

When a Worker has tail_consumers configured, Cloudflare appears to forward all node:diagnostics_channel messages to the Tail Worker as diagnosticsChannelEvents.

This also appears to make channel.hasSubscribers true for application libraries that use lazy diagnostics publishing.

In our case, upgrading to GraphQL.js 17 caused production-only errors:

Failed to publish diagnostics channel message:

The requests still returned 200 OK, but every GraphQL request produced runtime error logs.

Why this is surprising

We are not using OpenTelemetry, and our application does not subscribe to any diagnostics channels.

The only relevant production difference is that the deployed Worker has a Tail Worker configured.

Tail Workers are expected to observe/forward invocation output. They should not implicitly activate every library’s diagnostics_channel instrumentation.

Why this matters even with clone-safe payloads

The immediate error is caused by GraphQL.js publishing payloads containing objects that cannot be structured-cloned.

But even if those payloads were clone-safe, this behavior still creates unrequested overhead:

• extra library instrumentation work
• extra object allocation
• extra Tail event volume
• extra data transport/processing in the Tail Worker
• potentially more customer-paid logging/observability cost

tail_consumers should not silently change application behavior from “no diagnostics emitted” to “every lazy diagnostics publisher emits.”

Reproduction shape

1. Deploy a Worker with:
   • compatibility_flags = ["nodejs_compat"]
   • tail_consumers configured
   • a dependency that uses node:diagnostics_channel and gates publishing on channel.hasSubscribers
2. Make a request through that dependency.
3. Observe that diagnostics publishing activates in production even though the app never subscribed.

Concrete dependency observed: graphql@17.0.1, which creates graphql:* tracing channels.

Expected behavior

Tail Workers should not count as application-level subscribers for node:diagnostics_channel, or diagnostics forwarding should be explicitly configurable.

Possible fixes:

• do not make Tail Worker diagnostics forwarding affect channel.hasSubscribers
• add a tail_consumers option to disable diagnosticsChannelEvents
• add channel-name filtering, e.g. include/exclude graphql:*
• drop non-cloneable diagnostics payloads without emitting request-level errors
• document clearly that Tail Workers activate diagnostics channel publishers and may add overhead

Workaround

We patched GraphQL.js locally so its resolveDiagnosticsChannel() returns undefined, disabling its diagnostics channel integration entirely.

That avoids the production errors, but it is a brittle workaround and disables useful instrumentation for all environments.

Related issue

GraphQL.js issue: graphql/graphql-js#4834

This Cloudflare issue is not only about GraphQL payload clone failures. Even if GraphQL emits clone-safe payloads, Tail Workers still appear to activate lazy diagnostics publishers and create unrequested overhead.

[logaretm]

I think this should be an explicit toggle-on feature, one of the main advantages of diagnostic channels is they can have zero runtime overhead if there are no subscribers. This implies that all channels are activated which is a problem as more libraries are now adopting tracing channels, and will incur overhead the user may have not asked for.

Furthermore, the Spec has no limitation on the kind of values to be included in the message payload, in fact it is encouraged that subscribers mutate it and add their own data to it, in the observability space that would most certainly be span objects, scope, context, etc... all of which are not perfectly serializable.

[logaretm]

I dug a bit, and it seems like there is a partial fix with the compat flag date being: >=2026-05-19, at least it fixes the default case where there are no subscribers to the channel. However this wouldn't fix the underlying issue, having non-serializable values in the object should be perfectly fine.

It looks like the serialization logic here follows structuredClone, where it throws on methods, symbols, some classes, etc...

And these errors are being produced from here:

workerd/src/workerd/api/node/diagnostics-channel.c++

Lines 30 to 50 in 20bcc30

	auto& context = IoContext::current();
	KJ_IF_SOME(tracer, context.getWorkerTracer()) {
	js.tryCatch([&]() {
	jsg::Serializer ser(js,
	jsg::Serializer::Options{
	.omitHeader = false,
	});
	ser.write(js, jsg::JsValue(message.getHandle(js)));
	auto tmp = ser.release();
	JSG_REQUIRE(tmp.sharedArrayBuffers.size() == 0 && tmp.transferredArrayBuffers.size() == 0,
	Error,
	"Diagnostic events cannot be published with SharedArrayBuffer or "
	"transferred ArrayBuffer instances");
	tracer.addDiagnosticChannelEvent(
	context.getInvocationSpanContext(), context.now(), name.toString(js), kj::mv(tmp.data));
	}, [&](jsg::Value&& exception) {
	jsg::JsValue jsException(exception.getHandle(js));
	tracer.addException(context.getInvocationSpanContext(), context.now(), kj::str("Error"),
	kj::str("Failed to publish diagnostics channel message: ", jsException), kj::none);
	});
	}

I think we could one of the following:

• Silently drop the forwarded operation.
• Serialize the payload differently, using a best-effort mechanism to do it. Maybe a JSON-PARSE manner rather than structuredClone where non-serializable values are just dropped.

[jasnell]

Forwarding diagnostics channel to tail workers is intentional. I can absolutely update to better handle the non-serializable payload, however.

[logaretm]

Thanks, that would be great!

[yaacovCR]

I think we could one of the following:

• Silently drop the forwarded operation.
• Serialize the payload differently, using a best-effort mechanism to do it. Maybe a JSON-PARSE manner rather than structuredClone where non-serializable values are just dropped.

It seems like both options avoid the error message but neither lets graphql-js or other spec-compliant context entries that are not structured-clonable be actually usable in the tail worker.

It seems like we might need to work around the tail worker constraint with some sort of bridging API ideally that hits the channel goals of laziness/zero configuration outside APM, etc.

[JPeer264]

FWIW here is a minimal repro that showcases the error on the tailworker: https://github.com/JPeer264/sentry-repros/tree/issue-workerd-6861

[logaretm]

@yaacovCR There are a few things going on in combination that caused this issue:

• Tailworkers don't auto subscribe, they forward publish calls passively. So it doesn't actually incur the traced path automatically.
• There was a bug in workerd hasSubscribers implementation, it was a function not a getter so it caused the channel to publish events even tho there was no subscribers, this was corrected as of compat dates configured >= 2026-05-19.

So this wouldn't happen after that compat date unless someone subscribes to the channel.

So in this light, we can either do nothing as this will get fixed or we could do check if channel.hasSubscribers is a function, and if it is, call it, otherwise treat it as a getter. But this was a bug in the runtime.

[yaacovCR]

@logaretm it sounds like you are hinting that after the expected fixes, there would be no issue, even though our context is not structure-clonable?

Perhaps because there is some place already outside of graphql js where the context is/could be converted to something structure-clonable?

Or perhaps because I misunderstanding the failure mode/environment?

My understanding was that the apm in this context would live in the tail worker and so it would be missing information if we employed either of the two suggested options, but I am certainly and definitively out of my depth here.

:)

[logaretm]

after the expected fixes, there would be no issue, even though our context is not structure-clonable?

Yep, because this is affecting everyone right now that publishes similar objects and jasnell mentioned we can expect a fix for it. For the OP if they update their compat date right now they should be fine, unless they attach an APM that subscribes which isn't the case for them as the issue describes.

@jasnell is the fix for non-serializable values something we can expect soon?

[jasnell]

the immediate fix is that non-serializable values will result in a diagnostic channel log message with a message indicating that the payload could not be serialized rather than a logged exception. So you'll see that a diagnostic channel message was sent but the payload will be discarded. Only if THAT fails for some internal reason would an exception log occur.

We're still going to activate the diagnostic channel signals for tail workers by default. Separately (soon) I plan to add a compatibility flag that allows that to be selectively switched off.