getting null in response to api/v1/series

[certainmagic]

first, let me say that prometheus is great. along with grafana, we've just started using it and it is really helping us tell what's going on in our servers. so thanks!

i noticed that my grafana dashboards were choking on some of my template variable queries. it looks like the problem is that the response contains nulls.

here's the query:

http://prometheus:9090/api/v1/series?match[]=http_request_duration_seconds_bucket

here's a snippet of the response with almost all of the data elements removed:

{
    "data": [
        {
            "__name__": "http_request_duration_seconds_bucket",
            "instance": "1",
            "le": "20"
        },
        null,
        {
            "__name__": "http_request_duration_seconds_bucket",
            "instance": "2",
            "le": "0.1"
        }
    ],
    "status": "success"
}

i had this problem on 0.18.0. (prometheus, version 0.18.0 (branch: release-0.18, revision: f12ebd6)
i also tried running with 0.19.2 just in case it had been fixed recently. the problem continues in 0.19.2.

I asked for pointers in this thread.

Björn asked me to move my data aside and run again. That fixed the problem. I think that both of the servers that had this problem had run out of space earlier. I've also found that we were killing them ungracefully 90 seconds after sending a TERM to shut them down -- and that checkpoints are currently taking about 2 minutes, so they weren't shutting down cleanly. I've seen it run a crash recovery after that. We are addressing the disk space issues and have extended the timeout to let it checkpoint cleanly on shutdown.

Björn suggested that there's a good chance it was corrupted indices.

Björn asked me to file a bug suggesting that prometheus respond better to corruption by not returning the nulls and logging an error.

Please let me know if there's any more information I can provide.

thanks again!
ab

[juliusv]

Theory on how this can happen when the storage is corrupted:

Callstack:

• series() calls MetricsForLabelMatchers(): https://github.com/prometheus/prometheus/blob/0.19.2/web/api/v1/api.go#L265
• ...which calls metricForRange(): https://github.com/prometheus/prometheus/blob/0.19.2/storage/local/storage.go#L529
• ...which calls archivedMetric(): https://github.com/prometheus/prometheus/blob/0.19.2/storage/local/storage.go#L582
• ...which calls p.archivedFingerprintToMetrics.Lookup(): https://github.com/prometheus/prometheus/blob/0.19.2/storage/local/persistence.go#L1064
• ...which returns an ok boolean to indicate whether a fingerprint was found
• ...which is ignored by the archivedMetric() caller in https://github.com/prometheus/prometheus/blob/0.19.2/storage/local/persistence.go#L1064

[juliusv]

My storage fu currently doesn't go deep enough to immediately say on which level this should be filtered out. Whether archivedMetric() should raise an error if it cannot find a value, or whether metricForRange() should just filter out nil metrics.

[beorn7]

I'll look into it once I find time.

[beorn7]

With the changes that quarantine series that ran into an error, this should be solved.
Has this shown up again?

[cwarden]

I think I'm seeing the same problem in 1.5.1

$ curl -s -g 'http://prometheus:9090/api/v1/series?match[]={instance=~".*dev.*"}' | jq .
{
  "status": "success",
  "data": [
    null,
    null,
    ...
  ]
}
$ curl -s -XDELETE -g 'http://prometheus:9090/api/v1/series?match[]={instance=~".*dev.*"}' | jq .
{
  "status": "success",
  "data": {
    "numDeleted": 319
  }
}
$ curl -s -g 'http://prometheus:9090/api/v1/series?match[]={instance=~".*dev.*"}' | jq .
{
  "status": "success",
  "data": [
    null,
    null,
    ...
  ]
}
$ curl -s -XDELETE -g 'http://prometheus:9090/api/v1/series?match[]={instance=~".*dev.*"}' | jq .
{
  "status": "success",
  "data": {
    "numDeleted": 319
  }
}

[brian-brazil]

I'm going to presume this is resolved with 2.0, as the implementation all changed. If not, please let us know.

[certainmagic]

thanks! will do. :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.