Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make clock skew adjustment transparent #961

Closed
2 of 3 tasks
yurishkuro opened this issue Jul 26, 2018 · 14 comments
Closed
2 of 3 tasks

Make clock skew adjustment transparent #961

yurishkuro opened this issue Jul 26, 2018 · 14 comments

Comments

@yurishkuro
Copy link
Member

yurishkuro commented Jul 26, 2018

Requirement - what kind of business use case are you trying to solve?

It is currently not obvious in Jaeger UI is the span timestamps were adjusted for clock skew. @tedsuo wrote:

The issue was that displaying “manipulated” data, however well intentioned, led users to mistrust what they were seeing in some cases. “Is this 'real data' or is this ‘corrected data’?" was becoming a common question when looking at odd display patterns, or patterns that did not match other data sources. Given that we usually could not “correct” most scenarios properly but could only place the spans into a sort of “best guess” situation, we decided it would be better to avoid confusion and always show raw data, with visual annotations to point out where we felt the raw data was suspicious. So the main issue is trust, and retaining the ability to triage raw data. Another approach would be to give users a toggle to turn off the corrections so they can quickly see the raw data.

As an example, there were situations where clock correction was actually covering up issues, or subtly moving spans in a way which “looked right” but developers knew were impossible. For example, in one case there was a batch processing workload where the clock correction subtly moved things to make it look like the map/reduce phases were slightly overlapping in impossible ways. Other cases were where incorrect instrumentation lead to “incoceivable” child spans, but the source was not clock skew - that child span really was offset in a crazy way!

Problem - what in Jaeger blocks you from solving the requirement?

Jaeger clock skew adjustment is not recorded anywhere in the adjusted spans. UI does not make it obvious that the timestamps were adjusted.

Proposal - what do you suggest to solve the problem or improve the existing situation?

  • Clock skew adjuster should record the delta in the adjusted span
  • Jaeger UI should display a warning in the span details (and/or some other visual cue) that the span was adjusted
  • Jaeger UI should provide a toggle to view the unadjusted trace (the /api/trace endpoint already allows retrieving unadjusted trace) Add ability to display unadjusted trace in the UI jaeger-ui#197

Any open questions to address

How to do a visual cue that the span is adjusted.

@felippe-mendonca
Copy link

felippe-mendonca commented Aug 4, 2018

I had the same problem as exposed here. It took some time do figure out if the problem was mine or not. Because of that I think that could be interesting to have a mix between the second and third proposal: when displaying adjusted trace, warn the user about that.

@cameronbraid
Copy link

cameronbraid commented Jan 14, 2019

Is there any way to make the UI run without the clock skew adjuster ?

It is leading to incorrect representations of spans, shifting spans by hundreds of milliseconds in some cases.

screen shot 2019-01-14 at 7 04 40 pm

should be as follows

screen shot 2019-01-14 at 7 06 56 pm

@cameronbraid
Copy link

cameronbraid commented Jan 14, 2019

It turns out that the root cause of this was the root span (in blue in first image) finishing too soon. So the Clock Skew adjuster assumed that this was a clock skew issue masking the real problem.

@cameronbraid
Copy link

cameronbraid commented Jan 14, 2019

Actually , even with that fixed I get the same problem when the root span is on one server and the child spans on another server, I get the following

screen shot 2019-01-14 at 7 50 40 pm 1

The response-transformer span starts after the proxy-request span finishes and yet it always displays like this

@cameronbraid
Copy link

cameronbraid commented Jan 14, 2019

With the clock skew adjuster disabled it looks like this, which is exactly what I expect

screen shot 2019-01-14 at 8 00 49 pm

I don't want the UI to shift the spans in time to balance the 'latency' before and after it to be the same, that is just incorrect.

@yurishkuro
Copy link
Member Author

yurishkuro commented Jan 15, 2019

I agree. We already have an API call that returns "raw" trace, without adjustments. We just need a UI enhancement to use that API call to display the trace view.

@flixr
Copy link

flixr commented Nov 7, 2019

I'm also running into this problem...
Is there any way to turn off the adjustment globally?

@dhulchuk
Copy link

dhulchuk commented Nov 7, 2019

I'm also running into this problem...
Is there any way to turn off the adjustment globally?

We use next workaround for this problem:
You can set ip param in all traces to constant, so clock adjustment will be ignored for all new traces.

@dhulchuk
Copy link

dhulchuk commented Nov 7, 2019

Screen Shot 2019-11-07 at 16 46 07

bobrik added a commit to bobrik/jaeger that referenced this issue Feb 2, 2020
Time adjustment is confused to many people, as you can see in jaegertracing#961.

This change adds a warning in tags if we do any time adjustments,
so that it's at least clear that adjustments happened.
@bobrik
Copy link
Contributor

bobrik commented Feb 2, 2020

I opened #2052 to add a warning similar to what you can see in the comment above.

bobrik added a commit to bobrik/jaeger that referenced this issue Feb 2, 2020
Time adjustment is confused to many people, as you can see in jaegertracing#961.

This change adds a warning in tags if we do any time adjustments,
so that it's at least clear that adjustments happened.

Signed-off-by: Ivan Babrou <github@ivan.computer>
bobrik added a commit to bobrik/jaeger that referenced this issue Feb 2, 2020
Time adjustment is confusing to many people, as you can see in jaegertracing#961.

This change adds a warning if we do any time adjustments,
so that it's at least clear that adjustments happened.

Signed-off-by: Ivan Babrou <github@ivan.computer>
bobrik added a commit to bobrik/jaeger that referenced this issue Feb 2, 2020
Time adjustment is confusing to many people, as you can see in jaegertracing#961.

This change adds a warning if we do any time adjustments,
so that it's at least clear that adjustments happened.

Signed-off-by: Ivan Babrou <github@ivan.computer>
yurishkuro pushed a commit that referenced this issue Feb 2, 2020
Time adjustment is confusing to many people, as you can see in #961.

This change adds a warning if we do any time adjustments,
so that it's at least clear that adjustments happened.

Signed-off-by: Ivan Babrou <github@ivan.computer>
@bobrik
Copy link
Contributor

bobrik commented Feb 2, 2020

@yurishkuro, I think you can tick two first boxes in the issue description after #2052.

@yurishkuro
Copy link
Member Author

yurishkuro commented Mar 10, 2020

I believe this issue is resolved by #2052 and #2119.

@flixr
Copy link

flixr commented Mar 10, 2020

Just asking as it was part of the proposed solution: There are no plans anymore to add a toggle to view the unadjusted trace in the UI?

@yurishkuro
Copy link
Member Author

yurishkuro commented Mar 11, 2020

There is still an open ticket jaegertracing/jaeger-ui#197 for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants