Data lost after restart prometheus

[jofm]

Hi,

I'm starting using prometheus and I cannot evaluate if this is a issue or something that i'm missing. Also I can't find info about this.

I'm using Prometheus v1.5.0 with Pushgateway v0.3.1 both on linux Red Hat.
When injecting metrics on the pushgateway I need to specify a timestamp. Everything runs well until I restart prometheus service. After restart I notice that all values for each time period are just wrong, they simple don't match with the data before the restart.

Metric sample that go into pushgateway:

metric_two{metric_type="Modify",metric_sub_type="Synch"} 4 1485820800000
metric_one{metric_type="Hybrid2Pre",metric_sub_type="Hybrid2Pre"} 6 1485820800000
metric_two{metric_type="Hybrid2Pre",metric_sub_type="Hybrid2Pre"} 0 148582080000

I'm putting new metrics in pushgateway every hour and pushing with prometheus every 30 minutes (sync reasons with data collectors). Here's my config file:

global:
  scrape_interval:     30m 
  evaluation_interval: 30m 

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'orders_stats'
    static_configs:
      - targets: ['localhost:9091']

Don't know if this has anything to do with some kind of prometheus data aggregation that just go rogue because of the "forced" timestamp or some missing configuration.

Thanks in advance.

[brian-brazil]

A 30m scrape interval is going to run you into staleness issues. 2m is the practical maximum.

[jofm]

Thanks for the reply @brian-brazil

Does staleness issues affect the already collected data?

Below is a sample of my dashboard before and after restart prometheus for a better understanding.

The chart on the left uses influxdb as datasource and the one on the right is using prometheus. Both are sync.

Before restart (data between both graph are sync):

After restart (prometheus data went rogue):

KPI metric is calculated using the other two, under_5 and above_5 - under_5/(under_5+above_5) * 100
Metics values just change completely after restart.

Regards

[brian-brazil]

No, the data is still there but queries may have unexpected issues such as gaps.

[jofm]

Ok, but shouldn´t I experience that without a restart also?

[brian-brazil]

I'm presuming the restart is unrelated.

[rathohit311356]

How can we keep data file persistent ?
When we start Prometheus data gets deleted and new service start pulling data from scratch,
How can I configure to have my data persistent, I know about storage local path flag but no clue how to make my Prometheus data file persistent, please help advising

[krasi-georgiev]

@rathohit311356 sound like you have a lot of reading to do 🐰 ,
I would advise starting from the docs.
https://prometheus.io/docs/introduction/overview/

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.