Make clock skew adjustment transparent

[yurishkuro]

Requirement - what kind of business use case are you trying to solve?

It is currently not obvious in Jaeger UI is the span timestamps were adjusted for clock skew. @tedsuo wrote:

The issue was that displaying “manipulated” data, however well intentioned, led users to mistrust what they were seeing in some cases. “Is this 'real data' or is this ‘corrected data’?" was becoming a common question when looking at odd display patterns, or patterns that did not match other data sources. Given that we usually could not “correct” most scenarios properly but could only place the spans into a sort of “best guess” situation, we decided it would be better to avoid confusion and always show raw data, with visual annotations to point out where we felt the raw data was suspicious. So the main issue is trust, and retaining the ability to triage raw data. Another approach would be to give users a toggle to turn off the corrections so they can quickly see the raw data.

As an example, there were situations where clock correction was actually covering up issues, or subtly moving spans in a way which “looked right” but developers knew were impossible. For example, in one case there was a batch processing workload where the clock correction subtly moved things to make it look like the map/reduce phases were slightly overlapping in impossible ways. Other cases were where incorrect instrumentation lead to “incoceivable” child spans, but the source was not clock skew - that child span really was offset in a crazy way!

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger clock skew adjustment is not recorded anywhere in the adjusted spans. UI does not make it obvious that the timestamps were adjusted.

Proposal - what do you suggest to solve the problem or improve the existing situation?

• Clock skew adjuster should record the delta in the adjusted span
• Jaeger UI should display a warning in the span details (and/or some other visual cue) that the span was adjusted
• Jaeger UI should provide a toggle to view the unadjusted trace (the /api/trace endpoint already allows retrieving unadjusted trace) Add ability to display unadjusted trace in the UI jaeger-ui#197

Any open questions to address

How to do a visual cue that the span is adjusted.

[felippe-mendonca]

I had the same problem as exposed here. It took some time do figure out if the problem was mine or not. Because of that I think that could be interesting to have a mix between the second and third proposal: when displaying adjusted trace, warn the user about that.

[cameronbraid]

Is there any way to make the UI run without the clock skew adjuster ?

It is leading to incorrect representations of spans, shifting spans by hundreds of milliseconds in some cases.

should be as follows

[cameronbraid]

It turns out that the root cause of this was the root span (in blue in first image) finishing too soon. So the Clock Skew adjuster assumed that this was a clock skew issue masking the real problem.

[cameronbraid]

Actually , even with that fixed I get the same problem when the root span is on one server and the child spans on another server, I get the following

The response-transformer span starts after the proxy-request span finishes and yet it always displays like this

[cameronbraid]

With the clock skew adjuster disabled it looks like this, which is exactly what I expect

I don't want the UI to shift the spans in time to balance the 'latency' before and after it to be the same, that is just incorrect.

[yurishkuro]

I agree. We already have an API call that returns "raw" trace, without adjustments. We just need a UI enhancement to use that API call to display the trace view.

[flixr]

I'm also running into this problem...
Is there any way to turn off the adjustment globally?

[dhulchuk]

I'm also running into this problem...
Is there any way to turn off the adjustment globally?

We use next workaround for this problem:
You can set ip param in all traces to constant, so clock adjustment will be ignored for all new traces.

[dhulchuk]

[bobrik]

I opened #2052 to add a warning similar to what you can see in the comment above.