New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(prometheus_remote_write sink): Default timestamp to now #6823
Conversation
When encoding a metric to the remote write protocol, a timestamp field must be present. When presented with a metric without a timestamp, this sink encoded a timestamp of `0`, which is almost certainly not the correct value. This changes that default to the current time. Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Note, a second PR will be coming shortly to handle the other part of #6814. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good! I just have one question about the now
mocking.
Also, could we document this behavior in the external docs? I could see adding a trace
level log the timestamp is default to now to help debugging.
fn now(&mut self) -> i64 { | ||
*self | ||
.now | ||
.get_or_insert_with(|| Utc::now().timestamp_millis()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The naming of this function wouldn't indicate to me that it caches the value of now
. I'm wondering if we should just remove that caching.
Alternatively, I could see making this a top-level function and using cfg(test)
to define a mock implementation with a known value that the tests could depend on here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I figured it being both a simple optimization and a way of providing that all metrics in a batch used the same default timestamp. It has to be per-batch, as otherwise all metrics without a timestamp would share the same time again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That said, I am definitely open to renaming it to indicate the purpose better. I'll figure a more descriptive name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can see why it might be desirable for all metrics in the same batch to have the same default timestamp value, but is that an actual requirement? I was thinking it could just always use whatever Utc::now()
is, without the caching.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 the naming is definitely better now though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, it's not a requirement, just an optimization. On my desktop, Utc::now
is 4-6x as expensive as cloning a string, though amusingly it's faster than cloning on my MBP (which hints that MacOS has an aweful malloc implementation).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 I'm cool with it with the current naming. Thanks for making that change. Just to make sure I understand it, this is actually only applying to one timeseries though, isn't it?
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
* fix(prometheus_remote_write sink): Default timestamp to now When encoding a metric to the remote write protocol, a timestamp field must be present. When presented with a metric without a timestamp, this sink encoded a timestamp of `0`, which is almost certainly not the correct value. This changes that default to the current time. * Optimize calculating now to only once per request Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
When encoding a metric to the remote write protocol, a timestamp field
must be present. When presented with a metric without a timestamp, this
sink encoded a timestamp of
0
, which is almost certainly not thecorrect value. This changes that default to the current time.
Closes #6814
Signed-off-by: Bruce Guenter bruce.guenter@datadoghq.com