Deleted scrape target is always recreated

[piotrkochan]

Use case. Why is this important?

There is no really a way to stop prometheus from scraping previously defined target.

Bug Report

What did you do?

1. I removed scrape config from prometheus.yml
2. I ran:

curl -I -X POST -g 'localhost:9090/prometheus/api/v1/admin/tsdb/delete_series?match[]=up{instance="localhost:9100"}'
curl -I -X POST -g 'localhost:9090/prometheus/api/v1/admin/tsdb/clean_tombstones'

What did you expect to see?
I expect that if there is no config for this target and data then prometheus forget about that target, and do not try to scrape it in the future.

What did you see instead? Under which circumstances?

Prometheus still try to scrape this target. Even in the UI it is still visible:

Environment

• System information:

  Linux 4.4.0-130-generic x86_64

• Prometheus version:

prometheus, version 2.3.0 (branch: HEAD, revision: 290d71791a507a5057b9a099c9d48703d86dc941)
build user:       root@d539e167976a
build date:       20180607-08:46:54
go version:       go1.10.2

• Prometheus configuration file:

Just regular configuration without previously defined scrape_config entry.

[simonpasquier]

You need to reload the configuration using kill -SIGHUP <pid> for instance.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[piotrkochan]

@simonpasquier please reopen, because kill -SIGHUP does not work, as full prometheus restart.

[piotrkochan]

On the mailing list there is same issue and it is no resolved so I assume it is a bug.

https://groups.google.com/forum/#!searchin/prometheus-users/sighup%7Csort:date/prometheus-users/K4Q1ToxhwYw/EBNRvLKPBgAJ

[piotrkochan]

curl -I -X POST -g 'localhost:9090/prometheus/-/reload'
HTTP/1.1 200 OK

does not have any effect as well.

[piotrkochan]

I also tried to delete no only "up" but everything labeled:

curl -I -X POST -g 'localhost:9090/prometheus/api/v1/admin/tsdb/delete_series?match[]={instance="localhost:9100"}'

then reload, restart, service is still present...

[simonpasquier]

kill -SIGHUP ... works for me and I would be surprised if it was broken for everybody. Double check that you don't have a mismatch between the file used by Prometheus and the file you're modifying.

[marksugar]

Hello @piotrkochan , I don't know if you have solved this problem, because now I have encountered this problem and have not solved it.
If you have already solved it, can you tell me. In my environment prometheus runs in docker, curl -I -X POST -g 'localhost:9090/prometheus/-/reload' does not work

[piotrkochan]

I havent, target is still vislble W dniu śr., 26.12.2018 o 07:55 Mark <notifications@github.com> napisał(a):

…

Hello @piotrkochan <https://github.com/piotrkochan> , I don't know if you have solved this problem, because now I have encountered this problem and have not solved it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXy-TkV8FGdBqBn96e3QL7xPyEU5nbHks5u8x10gaJpZM4Xszk-> .

[marksugar]

@piotrkochan So how do you now handle alarm messages sent by hosts that no longer exist?
Best Regards

[piotrkochan]

Silence in alertmanager W dniu śr., 26.12.2018 o 10:24 Mark <notifications@github.com> napisał(a):

…

@piotrkochan <https://github.com/piotrkochan> So how do you now handle alarm messages sent by hosts that no longer exist? Best Regards — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXy-fyyI2bH1vsX_xiZXRxxIk0JVvI2ks5u80A4gaJpZM4Xszk-> .

[juliusv]

@piotrkochan

restart, service is still present...

If a full Prometheus restart does not remove the target, then it's certain that Prometheus is loading the wrong configuration file. Prometheus does not store a list of targets on disk by itself, so it would have no other source from which to re-create this target after restart.