Instrumentation best practice guide.

[tcolgate]

One of he best thing about the Prometheus site was the Best Practices documentation. This gave relatively product agnostic best practice tips for instrumenting code. Some was prom specific, but much was just good advice.

https://prometheus.io/docs/practices/instrumentation/

Similar advice for tracing is hard to come by.

e.g.

• how many spans is too many?
• span naming, (is cardinality a problem? we hit issue in jaeger)
• usage of logs (I've attempted to have all app logs within spanned code mirrored into spans, is that a good idea?)
• usage of message events
• what's appropriate for tags
• what's appropriate for propagated tags (once supported).

I'd be happy to contribute, perhaps source and edit advise, but I definitely have not had sufficient experience to know what is good or useful (we are just beginning our tracing journey).

[nikmd23]

I hear these questions from my customers a lot as well.

@tcolgate, I'd be happy to review and contribute additional leanings we come across to this if you want to take a first stab at it.

Additional "advice" to consider:

• How should I do sampling?
• What's the right sampling rate?
• How to handle "persistent connections" (WebSockets, etc)

Even if it isn't accepted as an OpenCensus doc, I think it's useful.