Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add staleness markers support #1526

Closed
hagen1778 opened this issue Aug 9, 2021 · 4 comments
Closed

Add staleness markers support #1526

hagen1778 opened this issue Aug 9, 2021 · 4 comments
Labels
enhancement New feature or request

Comments

@hagen1778
Copy link
Collaborator

hagen1778 commented Aug 9, 2021 •
edited

Is your feature request related to a problem? Please describe.
VictoriaMetrics and Prometheus staleness detection is different.

VictoriaMetrics calculates staleness threshold based on interval difference between datapoints timestamps (or scrape intervals).

Prometheus staleness logic is the following:

  • if a scrape fails then all time series from the previous scrape will be marked as stale;
  • if there's no samples within the 5 minutes before the evaluation, time series will be marked as stale.

VictoriaMetrics staleness detection behaves differently to Prometheus implementation for the following reasons:

  • staleness should work for all import protocols, and Prometheus remote write protocol is only one of them;
  • data may be ingested/backfilled into VictoriaMetrics, so "scrape" logic won't work;
  • to address staleness issues for series with intervals between data points exceeding 5 minutes.

However, because of the differences, query results for stale series between Prometheus and VictoriaMetrics may differ and result into discrepancies when using VictoriaMetrics as remote storage.

Describe the solution you'd like
To improve compatibility with Prometheus ecosystem, VictoriaMetrics TSDB and vmagent should support staleness markers.

Describe alternatives you've considered
Manual queries modifications to account for staleness and resets.
@hagen1778 hagen1778 added the enhancement New feature or request label Aug 9, 2021
@waldoweng
Copy link
Contributor

so this feature is in development now?

This was referenced Aug 9, 2021
Closed
Closed
@hagen1778
Copy link
Collaborator Author

It is not yet, but has a high priority.

@valyala valyala mentioned this issue Aug 11, 2021
Closed
@hagen1778 hagen1778 mentioned this issue Aug 13, 2021
Closed
valyala added a commit that referenced this issue Aug 13, 2021
@valyala
all: add support for Prometheus staleness markers
4401464 
Updates #1526
Updates #748
Updates #1509
Updates #1530
Updates #845
valyala added a commit that referenced this issue Aug 13, 2021
@valyala
all: add support for Prometheus staleness markers
c1f81f0 
Updates #1526
Updates #748
Updates #1509
Updates #1530
Updates #845
@valyala
Copy link
Collaborator

valyala commented Aug 13, 2021 •
edited

Support for Prometheus staleness markers has been added to VictoriaMetrics in the following commits:

Please give it a try before it will be included into the next release of VictoriaMetrics. See build instructions for single-node and build instructions for cluster version.

If the data is ingested into VictoriaMetrics by vmagent instead of Prometheus, then the vmagent must be built from the commit 4401464 according to build instructions, since the previous versions of vmagent do not generate Prometheus staleness marks when scrape targets disappear.

This was referenced Aug 14, 2021
Closed
Open
Closed
valyala added a commit that referenced this issue Aug 14, 2021
@valyala
app/vmselect/promql: properly handle Prometheus staleness marks in re…
af4a306 
…moveCounterResets functions

Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.

Updates #1526
valyala added a commit that referenced this issue Aug 14, 2021
@valyala
app/vmselect/promql: properly handle Prometheus staleness marks in re…
73d7b56 
…moveCounterResets functions

Prometheus stalenss marks shouldn't be changed in removeCounterResets. Otherwise they will be converted to an ordinary NaN values,
which couldn't be removed in dropStaleNaNs() function later. This may result in incorrect calculations for rollup functions.

Updates #1526
valyala added a commit that referenced this issue Aug 15, 2021
@valyala
app/vmselect/promql: drop staleness marks before calling rollupConfig.Do
5420c3d 
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.

Updates #1526
valyala added a commit that referenced this issue Aug 15, 2021
@valyala
app/vmselect/promql: drop staleness marks before calling rollupConfig.Do
113f0a8 
This allows dropping staleness marks only once and then calculate multiple rollup functions on the result.

Updates #1526
This was referenced Aug 15, 2021
Closed
Closed
@valyala
Copy link
Collaborator

valyala commented Aug 15, 2021

VictoriaMetrics and vmagent gained support for Prometheus staleness markers starting from v1.64.0. Closing the feature request as done.

@valyala valyala closed this as completed Aug 15, 2021
valyala added a commit that referenced this issue Aug 16, 2021
@valyala
lib/promscrape: stop scrapers for the removed targets before starting…
03c959f 
… scrapers for the added targets

This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).

Updates #1526
Updates #1530
Updates #748
Updates #1509
valyala added a commit that referenced this issue Aug 16, 2021
@valyala
lib/promscrape: stop scrapers for the removed targets before starting…
db34c40 
… scrapers for the added targets

This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).

Updates #1526
Updates #1530
Updates #748
Updates #1509
valyala added a commit that referenced this issue Aug 21, 2021
@valyala
lib/promscrape: reduce memory and CPU usage when Prometheus staleness…
d5622b3 
… tracking is enabled for metrics from deleted / disappeared scrape targets

Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.

This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.

Updates #1526
valyala added a commit that referenced this issue Aug 21, 2021
@valyala
lib/promscrape: reduce memory and CPU usage when Prometheus staleness…
67bc407 
… tracking is enabled for metrics from deleted / disappeared scrape targets

Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.

This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.

Updates #1526
@hagen1778 hagen1778 mentioned this issue Aug 30, 2021
Closed
@valyala valyala mentioned this issue Sep 20, 2021
Closed
@changshun-shi changshun-shi mentioned this issue Dec 20, 2023
Closed
3 tasks
valyala added a commit that referenced this issue Feb 8, 2024
@valyala
app/victoria-metrics: properly send staleness markers on victoriametr…
a354924 
…ics shutdown if -selfScrapeInterval > 0

Updates #943
Updates #1526
valyala added a commit that referenced this issue Feb 8, 2024
@valyala
app/victoria-metrics: properly send staleness markers on victoriametr…
582d431 
…ics shutdown if -selfScrapeInterval > 0

Updates #943
Updates #1526
valyala added a commit that referenced this issue Feb 8, 2024
@valyala
app/victoria-metrics: properly send staleness markers on victoriametr…
7fef8ba 
…ics shutdown if -selfScrapeInterval > 0

Updates #943
Updates #1526
valyala added a commit that referenced this issue Feb 8, 2024
@valyala
app/victoria-metrics: properly send staleness markers on victoriametr…
a2a218e 
…ics shutdown if -selfScrapeInterval > 0

Updates #943
Updates #1526
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@valyala @hagen1778 @waldoweng