RFC (pdk): standardized tracing api and core instrumentation

[vito-laurenza-zocdoc]

Summary

Currently, Kong has no tracing instrumentation in its core code. kong-plugin-zipkin exists, and is useful for getting a general sense of the timing of the different Nginx phases, but I'm seeing some unexplained latency that the plugin is not capturing.

I believe it would be hugely beneficial if Kong core took a lead in standardizing a tracing API within the PDK which could then be extended/implemented in plugins like the existing Zipkin one, or perhaps, a future OpenTelemetry plugin.

With a standardized API, Kong core could be instrumented in a concrete and fine-grained way. The API could be written in a way that the default implementation in core is a no-op, and only when a tracing plugin is enabled would the PDK tracing functions actually do anything.

This is mostly just a brain-dump of my thoughts at the moment. I'm happy to discuss further if there is interest in the idea. Thanks!

[vito-laurenza-zocdoc]

https://github.com/opentracing-contrib/nginx-opentracing

[hbagdi]

+1

There have been talks about doing the same with metrics in the PDK, where different implementations such as Prometheus, Datadog, statsD can pull metrics from the core.
Log serialization(f3653f8) recently made it's way into core.

Is this something you would be willing to contribute?

[jeremyjpj0916]

+1 to further enhanced tracing and latency explanations with core Kong. I have always thought a glaring feature missing is information on how long DB queries were taking on a given tx to help explain if the hits being seen were from lua code execution, nginx, or just waiting for db's to respond to given queries(less relevant for db-less setups but the lua code vs nginx trace exec time would still be ideal for those kinda deployments too).

[ludovic-pourrat]

+1

[secat]

+1

[isaac88]

@vito-laurenza-zocdoc did you make it works https://github.com/opentracing-contrib/nginx-opentracing in Kong ?

[vito-laurenza-zocdoc]

@vito-laurenza-zocdoc did you make it works https://github.com/opentracing-contrib/nginx-opentracing in Kong ?

No, I haven't had time to work on this.

[fkromer]

+1!!!

[blingenau]

+1
I recently debugged some latency issues in our deployment and to debug it I had to manually log things like ngx.ctx.KONG_ACCESS_TIME (as well as the other relevant context vars) and even added in my own additional ctx values for the before and after subphases. Having something built into kong to help with this sort of debugging would have been made this a much easier process

[ludovic-pourrat]

Related to this discussion #7331, we have pushed this plugin - https://github.com/matthyx/kong-elastic-apm

[chrisforrette]

Has anyone managed to get nginx-opentracing to work with Kong gateway?

I've been banging my head trying to get it working with Datadog trying to install the binary as well as using Docker multi-stage builds with no luck, as evidenced here: opentracing-contrib/nginx-opentracing#225

[mayocream]

+1

We are using skywalking-nginx-lua in Kong, but we'd like to migrate to an OpenTelemetry solution.

[chazdnato]

We have a few use cases where this kind of information would be useful. For example, in the rate limiting plugin we have set the redis connection timeout to a low value (such that if redis fails, we "fail open" faster for clients to still access resources). Knowing when this span in a trace is going above that threshold would be handy.

Another use case: we had an issue where the zipkin plugin was unable to talk to the upstream collector, and was consuming all the ngx.timers - this caused a cascade of failures in various subsystems and plugins that also use ngx.timers and it was hard to see where the problems were. Having a metric of timer usage could help us understand when we're reaching this limit again and scale differently.

In both cases, being able to have information generated in one plugin and passed to another (such as spans added to the kong.ngx.shared table) and then consumed and emitted via the zipkin or other metrics plugins, would have a lot of value.

[mateodelnorte]

I would very much like to see a plugin implemented that leverages https://github.com/opentracing-contrib/nginx-opentracing so users could choose their logging provider among those offered by the plugin.

[fffonion]

#8724 is merged, please hold for this new feature and new OpenTelemetry plugin in Kong 3.0 🎉 !