Add OpenCensus instrumentation #3303
Conversation
|
@jterry75 PTAL |
|
As a later change, I would like to start moving many log messages to use |
|
Build succeeded.
|
|
Build succeeded.
|
|
@crosbymichael - PTAL. The goal would be to do this: https://github.com/containerd/containerd/pull/3303/files#diff-6f891b9f28cfb10751fe182afa544a9cR87 on the ttrpc entry for a shim as well so the traceID/parentSpanID flows. Thanks! |
|
Build succeeded.
|
Codecov Report
@@ Coverage Diff @@
## master #3303 +/- ##
==========================================
- Coverage 44.6% 44.55% -0.06%
==========================================
Files 112 112
Lines 12180 12192 +12
==========================================
- Hits 5433 5432 -1
- Misses 5913 5925 +12
- Partials 834 835 +1
Continue to review full report at Codecov.
|
|
link to #3057 |
|
This is very useful! Is it possible to make this a plugin? So that:
|
|
@Random-Liu - I think it could be a config option if necessary but I don't see how it could be a plugin. We need to add the gRPC option at containerd server init. I don't think this can take a plugin dependency can it? |
|
@jterry75 At least the exporter seems to be a plugin to me. |
|
@Random-Liu @jterry75 how do you feel about build tag? |
|
@mxpv Yeah, if we make it a plugin we need a build tag. |
| @@ -62,15 +63,26 @@ func WithLogger(ctx context.Context, logger *logrus.Entry) context.Context { | |||
| } | |||
|
|
|||
| // GetLogger retrieves the current logger from the context. If no logger is | |||
| // available, the default logger is returned. | |||
| // available, the default logger is returned. If the context has a span | |||
| // associated with it, add correlating information to the returned logger. | |||
| func GetLogger(ctx context.Context) *logrus.Entry { | |||
There was a problem hiding this comment.
Is it going to be confusing hiding the tracing functionality in the log pkg and GetLogger function?
|
@dmcgowan also prefers this implemented as a set of plugins for the various functionality. So I am going to look at breaking this change into several plugins. |
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
This change has three parts: - Add the OC GRPC handler to the GRPC server. This causes a new OC span to be started and associated with the context for every incoming GRPC request. If the request metadata includes existing span context, the new span will be parented to the existing span. - Add a small exporter to log completed span data to Logrus. OC supports the concept of exporters which send the data for completed spans to an external system for storage and tracking. I created a logrusExporter which simply logs some of the span data to Logrus. While this obviously isn't a complete distributed tracing system, it allows span data to be easily stored in log files and seen on the console. It also works nicely with the existing use of Logrus hooks to send log data to other systems (such as ETW on Windows). Long-term, we should also add the OC agent exporter, which sends the span data to the OC agent running on the same system. This is the recommended path for exporting spans. However, since this requires an additional binary running on the system, I think there is value in keeping the logrusExporter as a light-weight alternative. - Modify log.GetLogger to add span correlation data to the logrus.Entry. When a Logrus entry is returned from log.GetLogger, if the context also contains a saved span, then information is added to the entry to allow correlating the log messages with the current span. This will cause all messages logged via log.G(ctx) to contain additional fields for trace ID, span ID, and sampling status. This does make these messages more verbose, but will be very useful when correlating logs to spans. Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
|
Build succeeded.
|
This change has three parts:
Add the OC GRPC handler to the GRPC server.
This causes a new OC span to be started and associated with the
context for every incoming GRPC request. If the request metadata
includes existing span context, the new span will be parented to the
existing span.
Add a small exporter to log completed span data to Logrus.
OC supports the concept of exporters which send the data for completed
spans to an external system for storage and tracking. I created a
logrusExporter which simply logs some of the span data to Logrus.
While this obviously isn't a complete distributed tracing system, it
allows span data to be easily stored in log files and seen on the
console. It also works nicely with the existing use of Logrus hooks
to send log data to other systems (such as ETW on Windows).
Long-term, we should also add the OC agent exporter, which sends the
span data to the OC agent running on the same system. This is the
recommended path for exporting spans. However, since this requires
an additional binary running on the system, I think there is value in
keeping the logrusExporter as a light-weight alternative.
Modify log.GetLogger to add span correlation data to the logrus.Entry.
When a Logrus entry is returned from log.GetLogger, if the context
also contains a saved span, then information is added to the entry to
allow correlating the log messages with the current span.
This will cause all messages logged via log.G(ctx) to contain
additional fields for trace ID, span ID, and sampling status. This
does make these messages more verbose, but will be very useful when
correlating logs to spans.
Signed-off-by: Kevin Parsons kevpar@microsoft.com