Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(prometheus_remote_write sink): Default timestamp to now #6823

Merged
merged 3 commits into from Mar 23, 2021

Conversation

bruceg
Copy link
Member

@bruceg bruceg commented Mar 19, 2021

When encoding a metric to the remote write protocol, a timestamp field
must be present. When presented with a metric without a timestamp, this
sink encoded a timestamp of 0, which is almost certainly not the
correct value. This changes that default to the current time.

Closes #6814

Signed-off-by: Bruce Guenter bruce.guenter@datadoghq.com

When encoding a metric to the remote write protocol, a timestamp field
must be present. When presented with a metric without a timestamp, this
sink encoded a timestamp of `0`, which is almost certainly not the
correct value. This changes that default to the current time.

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
@bruceg bruceg added type: bug A code related bug. sink: prometheus_remote_write Anything `prometheus_remote_write` sink related labels Mar 19, 2021
@bruceg bruceg self-assigned this Mar 19, 2021
@bruceg bruceg requested a review from a team March 19, 2021 17:04
@bruceg
Copy link
Member Author

bruceg commented Mar 19, 2021

Note, a second PR will be coming shortly to handle the other part of #6814.

Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! I just have one question about the now mocking.

Also, could we document this behavior in the external docs? I could see adding a trace level log the timestamp is default to now to help debugging.

fn now(&mut self) -> i64 {
*self
.now
.get_or_insert_with(|| Utc::now().timestamp_millis())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming of this function wouldn't indicate to me that it caches the value of now. I'm wondering if we should just remove that caching.

Alternatively, I could see making this a top-level function and using cfg(test) to define a mock implementation with a known value that the tests could depend on here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured it being both a simple optimization and a way of providing that all metrics in a batch used the same default timestamp. It has to be per-batch, as otherwise all metrics without a timestamp would share the same time again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said, I am definitely open to renaming it to indicate the purpose better. I'll figure a more descriptive name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see why it might be desirable for all metrics in the same batch to have the same default timestamp value, but is that an actual requirement? I was thinking it could just always use whatever Utc::now() is, without the caching.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 the naming is definitely better now though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's not a requirement, just an optimization. On my desktop, Utc::now is 4-6x as expensive as cloning a string, though amusingly it's faster than cloning on my MBP (which hints that MacOS has an aweful malloc implementation).

Copy link
Member

@jszwedko jszwedko Mar 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I'm cool with it with the current naming. Thanks for making that change. Just to make sure I understand it, this is actually only applying to one timeseries though, isn't it?

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
@bruceg bruceg merged commit 14b778e into master Mar 23, 2021
@bruceg bruceg deleted the prometheus-remote-write-timestamp-now branch March 23, 2021 21:10
jszwedko pushed a commit that referenced this pull request Mar 30, 2021
* fix(prometheus_remote_write sink): Default timestamp to now

When encoding a metric to the remote write protocol, a timestamp field
must be present. When presented with a metric without a timestamp, this
sink encoded a timestamp of `0`, which is almost certainly not the
correct value. This changes that default to the current time.

* Optimize calculating now to only once per request

Signed-off-by: Bruce Guenter <bruce.guenter@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sink: prometheus_remote_write Anything `prometheus_remote_write` sink related type: bug A code related bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prometheus_remote_write writes metrics with a timestamp of 0 if the metrics source has no timestamp
2 participants