python prometheus_client guage metrics are not saved in prometheus

[psri6891]

import prometheus_client as prom, Guage
import random
import time


pool={'pool_name': 'testing-prom-tool','member_name': 'promtest', 'mem_port': '443', 'mem_address': 'xx.xx.xx.xx', 'mem_state': 'down'}
# Create a metric to track time spent and requests made.
REQUEST_TIME = prom.Summary('request_processing_seconds', 'Time spent processing request')


# Decorate function with metric.
@REQUEST_TIME.time()
def process_request():
    time.sleep(1)


if __name__ == '__main__':
    #                          name documentation label names ###  I was able to populate all the keys from the dictionary using the pool. keys())
    f5_prom_test = prom.Guage('f5_test','f5_node_status',('pool_name','member_name','mem_port','mem_address','mem_state'))
    prom.start_http_server(1234)
While True:
   process_request()
        f5_prom_test.labels(pool.get('pool_name'),pool.get('member_name'),pool.get('mem_port'),pool.get('mem_address'),pool.get('mem_state'))
#f5_prom.labels(**pool), this works as well

I can see metrics are registered when I curl -K http://localhost:1234. Somehow the metrics are not saved in the Prometheus. I can't view the data whenever I stop the python script in grafana, and there is no historical data held in the Prometheus tsdb to view on the Prometheus web URL

curl -K http://localhost:1234

f5_test_created{mem_address="xx.xx.xx.xxx",mem_name="test-server",pool_name"=testpool",mem_port="5443",mem_state="down"} 1.658982617711136e+09

Here is my prometheus.yml for custom python Prometheus collector for pool data

  - job_name: 'python-exporter
    scrape_interval: 5s
    static_configs:
      - targets: ['hostname:1234']

I can view the data only when I use the python script; later, that data is not saved in Prometheus. I'm not using any custom registry
, How to save registered data in /metrics to Prometheus using prometheus_client? I already changed my retention period of Prometheus tsdb

[csmarchbanks]

Hello!

If you are able to see the metrics via curl then you are using the prometheus_client library properly. Are there any errors in the prometheus UI under Status > Targets? Your python script will have to be running for Prometheus to be able to scrape it.

[psri6891]

I can see the target is up in Prometheus UI. I ran the python script for like an hour, and the grafana dashboard only shows the data when the script is running, but why is that data not saved in Prometheus tsdb? Here's what I'm trying to achieve, I'm pulling f5 load balancer data using python SDK and I want to save that data to prometheus tsdb by writing a prometheus exporter. I have been able to write the exporter but I think I am missing something here maybe the registry that's why the data is not saved in prometheus?

following script output containers metric_name and labels and when I see the output in the grafana it shows the latest timestamp, the way prometheus scrapes the metrics from the target and while the exporter is running, grafana dashboard shows data, once the exporter is stopped the data is disappeared. How come prometheus not saving the metrics while the prometheus exporter is running?
f5_test_created{mem_address="xx.xx.xx.xxx",mem_name="test-server",pool_name"=testpool",mem_port="5443",mem_state="down"}

import prometheus_client as prom, Guage
import random
import time


pool={'pool_name': 'testing-prom-tool','member_name': 'promtest', 'mem_port': '443', 'mem_address': 'xx.xx.xx.xx', 'mem_state': 'down'}
# Create a metric to track time spent and requests made.
REQUEST_TIME = prom.Summary('request_processing_seconds', 'Time spent processing request')


# Decorate function with metric.
@REQUEST_TIME.time()
def process_request():
    time.sleep(1)


if __name__ == '__main__':
    #                          name documentation label names ###  I was able to populate all the keys from the dictionary using the pool. keys())
    f5_prom_test = prom.Guage('f5_test','f5_node_status',('pool_name','member_name','mem_port','mem_address','mem_state'))
    prom.start_http_server(1234)
While True:
   process_request()
        f5_prom_test.labels(pool.get('pool_name'),pool.get('member_name'),pool.get('mem_port'),pool.get('mem_address'),pool.get('mem_state'))
#f5_prom.labels(**pool), this works as well

[csmarchbanks]

Hmm, that is strange, Prometheus should be saving the metric, and if it was visible in Grafana then it should be in the TSDB for that time range. If the script is stopped then I would expect the metrics to stop being reported and be marked stale, but if you zoom to the time range when the script was running they should still be there.

Would you be able to provide a bit more information around your Prometheus version and config? Both any non-default cli flags and relevant parts of the config file. Any warning/error logs from Prometheus would also be helpful.

Since you can see metrics in Prometheus/Grafana some of the time that sounds like your exporter is working properly at least.

[psri6891]

I tried zooming into the time range but nothing in grafana; I looked for the data in Prometheus UI nothing after the exporter was stopped. Where do I look for Prometheus logs? I don't see any errors in Prometheus UI. Let me know if I can turn any debugging option I can turn on and monitor logs while the exporter is running and stopped.

prometheus, version 2.27.1 (branch: HEAD, revision: db7f0bcec27bd8aeebad6b08ac849516efa9ae02)
build user: root@fd804fbd4f25
build date: 20210518-14:17:54
go version: go1.16.4
platform: Linux/amd64

prometheus.yml for the python-exporter

  - job_name: 'python-exporter
    scrape_interval: 10m
    static_configs:
      - targets: ['hostname:1234']

linux .service file contents

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/Prometheus \
    --config.file /prometheus/prometheus.yml \
    --storage.sub.retention.time=30d

[Install]
WantedBy=multi-user.target

[psri6891]

My python_exporter data, which is running on http://hostname_where_exporter_running:8080/metrics, updates every 10mins; that's why I chose a 10min scrape interval. When prometheus scrapes the endpoint http://hostname_where_exporter_running:8080/metrics using job_name and scrape interval in the prometheus.yml file when I update my scrape_interval to 1m, does that mean prometheus will scrape the endpoint every 1m and save that data into the Prometheus in 1 min intervals(10 entries in 10mins with different timestamps but same data points will be written (the exporter data updates on 10 mins interval) to Prometheus tsdb?. Does scrape interval duration decide when data should be reported to prometheus tsdb? How long does the exporter need to run to save the metrics to prometheus?

  - job_name: 'python-exporter
    scrape_interval: 1m
    static_configs:
      - targets: ['hostname:1234']

[csmarchbanks]

Correct, a scrape interval of 1m will mean the same value is stored in the database for ten different timestamps. Repeating values compress very well so very little extra data will be used compared to scraping every 10 minutes. Prometheus will initiate a scrape and save the collected data to the tsdb during each scrape every minute.

Ideally the exporter would run continuously, but as long as you run it for more than a minute there should be at least one datapoint in your TSDB. Depending on the queries you are using you may need more than one data point (for example rate queries) at which point it will need to run for at least a few minutes.

[psri6891]

I looked at the last 30 days of data on the Prometheus UI browser by using pool_details{name=~"testing-pool-purpose"}[30d], and I do see the data is saved to Prometheus; the reason for information is not shown in grafana is due to I used rate queries and instant on grafana table option which only shows the latest timestamp. When I turned off instant, I could see the data in grafana; I need to structure my queries better to get the data in grafana, something like show time in 10 mins intervals with the label's name. I changed my scrape interval duration to 1 minute(globally and deleted local job-scrape interval for all jobs in .config file), but I still see the time difference of 15 secs in the grafana dashboard instead of 1 min. Why is that so?

[csmarchbanks]

Grafana is likely querying with a resolution of 15s at which point Prometheus will return the previous value for those intermediary timestamps. That is by design and not something to worry about. I think you can add a minimum step of 1m in Grafana if you want though.

[psri6891]

Thanks, @csmarchbanks, for answering all my questions.