Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CURL sometimes not working #5425

Closed
Shanee33 opened this Issue Apr 1, 2019 · 3 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@Shanee33 @simonpasquier
@Shanee33
Copy link

Shanee33 commented Apr 1, 2019

Proposal

When I make changes to either the config file or alert config file and do a CURL XPOST command prometheus does not change, but when I restart the service it works

Bug Report

In the case I was talking about I had made a rule previous and wanted to another, so I made the changes in the relevant alerting file. I have added a new rule and ran a CURL and nothing changed. I deleted the rule and ran the CURL and the rule was still there, even after refreshing the page I could see that the rule was still active as it was showing how long it takes to check the rule.

After I restarted the promethus service the changes took effect, but I dont want to do this as we have gaps in our graphs

To clarify I have the --web.enable-lifecycle flag running and CURL does work most over the time when I have made changes to targets and jobs etc

What did you expect to see?
Changes to rules

What did you see instead? Under which circumstances?
No changes

  • System information:

Linux 3.10.0-957.5.1.el7.x86_64 x86_64

  • Prometheus version:

2.2.1

  • Alertmanager version:

0.16.1

  • Prometheus configuration file:
insert configuration here
  • Alertmanager configuration file:
insert configuration here (if relevant to the issue)
global:
  scrape_interval:     5s
  scrape_timeout:      5s

rule_files:
  - alert.rules.yml

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "rbiqhsmgtt106v:9093"

scrape_configs:
# Prometheus
  - job_name: 'Prometheus'
    file_sd_configs:
     - files :
       - /prometheus/targets/node/targetPrometheus.yml

* Logs:

Apr 01 00:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-03-31T23:00:00.490363923Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554062
Apr 01 00:00:04 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-03-31T23:00:04.242286866Z caller=head.go:348 component=tsdb msg="head GC completed" duration=66.815297ms
Apr 01 00:00:14 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-03-31T23:00:14.041375242Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=9.79
Apr 01 02:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T01:00:00.490194888Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554069
Apr 01 02:00:03 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T01:00:03.331197433Z caller=head.go:348 component=tsdb msg="head GC completed" duration=70.325764ms
Apr 01 02:00:04 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T01:00:04.881979239Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.55
Apr 01 04:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T03:00:00.490598333Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554076
Apr 01 04:00:05 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T03:00:05.418825742Z caller=head.go:348 component=tsdb msg="head GC completed" duration=61.284849ms
Apr 01 04:00:21 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T03:00:21.166387847Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=15.7
Apr 01 04:00:21 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T03:00:21.69927629Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=15540552
Apr 01 06:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T05:00:00.490270501Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554084
Apr 01 06:00:03 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T05:00:03.168056786Z caller=head.go:348 component=tsdb msg="head GC completed" duration=85.967098ms
Apr 01 06:00:07 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T05:00:07.63863982Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=4.470
Apr 01 08:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T07:00:00.490323067Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554091
Apr 01 08:00:05 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T07:00:05.662190923Z caller=head.go:348 component=tsdb msg="head GC completed" duration=72.777358ms
Apr 01 08:00:07 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T07:00:07.927198925Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=2.26
Apr 01 10:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T09:00:00.490520717Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554098
Apr 01 10:00:03 X.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T09:00:03.556550191Z caller=head.go:348 component=tsdb msg="head GC completed" duration=68.995772ms
Apr 01 10:00:07 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T09:00:07.692392808Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=4.13
Apr 01 10:00:08 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T09:00:08.239169603Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1554076
Apr 01 10:00:16 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T09:00:16.752392069Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1554033
Apr 01 12:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T11:00:00.490133365Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554105
Apr 01 12:00:05 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T11:00:04.96077008Z caller=head.go:348 component=tsdb msg="head GC completed" duration=68.577108ms
Apr 01 12:00:12 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T11:00:12.643126979Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=7.68
Apr 01 14:00:00 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T13:00:00.490319342Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1554112
Apr 01 14:00:03 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T13:00:03.466213136Z caller=head.go:348 component=tsdb msg="head GC completed" duration=59.749617ms
Apr 01 14:00:04 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T13:00:04.981059029Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.51
Apr 01 15:11:17 XX.b2b.regn.net systemd[1]: Stopping Prometheus Server...
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=warn ts=2019-04-01T14:11:17.284098614Z caller=main.go:374 msg="Received SIGTERM, exiting gracefully..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.284191585Z caller=main.go:398 msg="Stopping scrape discovery manager..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.284241152Z caller=main.go:411 msg="Stopping notify discovery manager..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.2842567Z caller=main.go:432 msg="Stopping scrape manager..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.284290992Z caller=main.go:394 msg="Scrape discovery manager stopped"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.285437836Z caller=main.go:407 msg="Notify discovery manager stopped"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.290207264Z caller=main.go:426 msg="Scrape manager stopped"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.303304777Z caller=manager.go:460 component="rule manager" msg="Stopping rule manager..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.303344556Z caller=manager.go:466 component="rule manager" msg="Rule manager stopped"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.303369253Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.303387115Z caller=main.go:573 msg="Notifier manager stopped"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[18066]: level=info ts=2019-04-01T14:11:17.303607645Z caller=main.go:584 msg="See you next time!"
Apr 01 15:11:17 XX.b2b.regn.net systemd[1]: Stopped Prometheus Server.
Apr 01 15:11:17 XX.b2b.regn.net systemd[1]: Started Prometheus Server.
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.480972645Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=H
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.481058106Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=201
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.481082712Z caller=main.go:222 host_details="(Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri F
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.481107506Z caller=main.go:223 fd_limits="(soft=1024, hard=4096)"
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.487404373Z caller=main.go:504 msg="Starting TSDB ..."
Apr 01 15:11:17 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:17.511546951Z caller=web.go:382 component=web msg="Start listening for connections" address=
Apr 01 15:11:40 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:40.666490118Z caller=main.go:514 msg="TSDB started"
Apr 01 15:11:40 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:40.666557225Z caller=main.go:588 msg="Loading configuration file" filename=/prometheus/prome
Apr 01 15:11:40 XX.b2b.regn.net prometheus[23627]: level=info ts=2019-04-01T14:11:40.694608342Z caller=main.go:491 msg="Server is ready to receive web requests."
@simonpasquier

This comment has been minimized.

Sign in to view
Copy link
Member

simonpasquier commented Apr 1, 2019

Duplicate of #5195
I would still recommend to upgrade to a newer version of Prometheus as 2.2.1 is quite old and unmaintained.

@simonpasquier simonpasquier closed this Apr 1, 2019

@Shanee33

This comment has been minimized.

Sign in to view
Copy link
Author

Shanee33 commented Apr 1, 2019

I have now upgraded to 2.8.1 and am still having the same issue
@simonpasquier

This comment has been minimized.

Sign in to view
Copy link
Member

simonpasquier commented Apr 2, 2019

This is expected as #5195 is still unresolved. For now you'd have to restart Prometheus when removing rules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.