Allow request traceability via distributed tracing

[chadell]

As ...

Patti - Platform Admin

I want ...

to have more detailed visibility of Nautobot operations to better understand how the different steps in the process of a request are executed.

So that ...

I could understand where in the processing of a request more time is spent, and troubleshoot problems more effectively.

For instance, the OpenTelemetry standard, leveraging Django instrumentation could be a good starting point because it allows easier integration and a standard definition for Traces (and metrics and logs).
A visualization like this could help understand the related interactions of a request:

I know this is done when...

Nautobot offers traces for the most significant operations, and also allows other apps in the ecosystem to leverage it.

Optional - Feature groups this request pertains to.

• Automation
• Circuits
• DCIM
• IPAM
• Misc (including Data Sources)
• Organization
• Plugins (and other Extensibility)
• Security (Secrets, etc)
• Image Management
• UI/UX
• Documentation
• Other (not directly a platform feature)

Database Changes

No response

External Dependencies

• opentelemetry-sdk
• opentelemetry-instrumentation-django

[lampwins]

I think you are posing this request in terms of tracing a flow within Nautobot/Job/etc. I would also like to consider this in the context of Nautobot participating in a larger distributed system/workflow. I.e. another system creates some data in Nautobot and provides a trace/span ID for tracking.

[chadell]

Yes, I agree. My proposal was about the big picture (including the Nautobot internal one)

[glennmatthews]

• https://www.datadoghq.com/knowledge-center/distributed-tracing/
• https://opentelemetry.io/docs/instrumentation/python/
• https://opentelemetry-python.readthedocs.io/en/stable/examples/django/README.html
• https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/django/django.html
• https://opentelemetry.io/docs/specs/otel/overview/

[chadell]

Yes, that's exactly it.

[gsnider2195]

How much of this request would be covered by #5169? Or is this simply requesting that some type of identifier can be provided in web requests and Nautobot retains that identifier throughout the request's lifetime?

[chadell]

It is about taking and identifier and pass through other intergations to create an end-to-end traceability

[gsnider2195]

@chadell would this be a sufficient definition of done for this issue?

• Nautobot is able to accept an arbitrary identifier in an HTTP request
• Nautobot displays that identifier in all logging messages
• Any ObjectChange instances created by this request will store the identifier
• All API responses include this identifier?
• All UI responses include this identifier in an html comment?

[lampwins]

https://github.com/open-telemetry/opentelemetry-python-contrib

[chadell]

Yes, this is what I had in mind as a potential solution, OpenTelemetry.
AFAIK, @nlgotz has been exploring it.

[nlgotz]

I started testing adding in OpenTelemetry in #5295 mainly to get more familiar with Tempo/tracing.

There's still a lot of work/testing needed before it's ready though. The PR so far has the ability to do a trace for normal page views and also running a job. The job run shows everything related to Django, Celery, Postgres, and Redis.