Deletion of metrics through API no longer working in 2.0

[keithf4]

Using methods that were successful in Prometheus 1.8 and as suggested in previous issues (#2356), we're trying to remove specific job data from the prometheus database using the API. This seems to no longer work in 2.0. Not sure if it's relevant, but I did make sure that the admin api is turned on (--web.enable-admin-api is set at startup)

What did you do?

curl -XDELETE -g "http://localhost:9090/api/v1/series?match[]={job='devdb5'}"

What did you expect to see?

Removal of all metrics with jobname devdb5

What did you see instead? Under which circumstances?

Returned with error:

{"status":"error","errorType":"internal","error":"not implemented"}

Environment

CentOS7

• System information:

$ uname -srm
Linux 3.10.0-693.11.1.el7.x86_64 x86_64

• Prometheus version:

$ prometheus --version
prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

[gouthamve]

Hi, deletions are now in master and will be available with docs in the next release.

For now, in 2.0, see: https://groups.google.com/forum/#!msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ

[keithf4]

Thanks!

[keithf4]

So tried this and doesn't seem to be actually deleting anything despite returning 200/OK

$ curl -sw "\n%{http_code}\n" -X POST -d "{
  \"matchers\": [{
  \"type\": \"EQ\",
  \"name\": \"job\",
  \"value\": \"devdb2\"
  }]
}" -H "Content-Type: application/json" http://localhost:9090/api/v2/admin/tsdb/delete_series
{}
200

Getting a list of all the label values using curl http://localhost:9090/api/v1/label/job/values shows that the job data label still exists.

[keithf4]

Actually, looks like the data was deleted, but the job label is still in the system. The old delete method would get rid of the label as well once all the data was gone. There another API interface to do that?

[gouthamve]

So we do tombstones and not actually delete the data. Even if you select the series with the joblabel, then nothing will turn up.

We do have an endpoint which deletes the tombstones and cleans up the data and index but that is in master and not 2.0.

[keithf4]

Ok. Coworker did find this looking around in the github repo

https://github.com/prometheus/prometheus/blob/master/documentation/dev/api/swagger.json

But when I tried using the clean_tombstones command I was getting 404's. I guess that would be why!

Main issue we're running into is with Grafana and the "label_values()" function. This is still apparently picking up these tombstones so we're having old systems still show up in the drop down lists. So sounds like we're going to have to wait for next release before we can clean these up?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.