Prometheus Crash on Querying a Specific Metric

[muralikanagala]

Hello There,

We are using below version of Prometheus.
prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc) build user: root@149e5b3f0829 build date: 20180314-14:15:45 go version: go1.10
We have recently added a new metric for netstat connections and gathering it using textfile_collector on wmi_exporter. We are able to see the metrics on wmi_exporter but while querying them on Prometheus dashboard Prometheus service is crashing with below error.

Sep 28 17:30:06 prometheus[7297]: level=warn ts=2018-09-28T21:30:06.653633148Z caller=head.go:320 component=tsdb msg="unknown series references in WAL samples" count=6

So I tried deleting that time-series using the API but getting no response followed by another crash of Prometheus.

curl -vv --silent -H "Content-Type: application/json" -X POST -d '{"matchers":[{"type":"EQ","name":"__name__","value":"netstat_connections"}]}' 'http://localhost:9090/api/v2/admin/tsdb/delete_series'
* About to connect() to localhost port 9090 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 9090 (#0)
> POST /api/v2/admin/tsdb/delete_series HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:9090
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
>
* upload completely sent off: 76 out of 76 bytes
* Empty reply from server
*Connection #0 to host localhost left intact`

Can someone help me fixing this issue?

Thank you,
Murali

[gouthamve]

Hi, could you try the latest version 2.4.2. There were some issues with the delete API that were already fixed.

[muralikanagala]

Thank you! We will try the upgrade.