Remote Read runtime error with /api/v1/series

[steve-exley]

Bug Report

When using Remote Read between two Prometheus 2.3.0 systems. Using the /api/v1/series causes Prometheus to drop the connection and generates an error in the logs:

For example:
http://y.y.y.y:9090/api/v1/series?match[]=scrape_duration_seconds

I have tested removing the remote_read configuration and then Prometheus behaves as expected and returns metrics from /api/v1/series requests.

I have also tested /api/v1/query and /api/v1/query_range requests and these return metrics as expected including the remote ones. The issue seems to only occur on /api/v1/series requests which are important in my use case because Grafana uses them to generate lists.

What did you expect to see?

When making a request to the prometheus
parent: http://y.y.y.y:9090/api/v1/series?match[]=scrape_duration_seconds
I expected to get a HTTP: 200 response with metrics.

What did you see instead? Under which circumstances?
Prometheus closed the connection and logged:
prometheus: level=error ts=2018-07-04T06:42:50.15051076Z caller=stdlib.go:89 component=web caller="http: panic serving x.x.x.x:49204" msg="runtime error: invalid memory address or nil pointer dereference"

Environment

• System information:

  Linux 3.10.0-862.3.3.el7.x86_64 x86_64

• Prometheus version:

  2.3.0 (Both Prometheus instances)

• Prometheus configuration file:

Prometheus parent instance y.y.y.y:

global:
  evaluation_interval: 15s
  scrape_interval: 30s
  scrape_timeout: 10s
  external_labels:
    {}

remote_read:
  - read_recent: true
    remote_timeout: 30s
    url: http://x.x.x.x:9090/api/v1/read

rule_files:
  - /etc/prometheus/rules/*.rules

scrape_configs:
  ...

Prometheus client instance x.x.x.x:

global:
  evaluation_interval: 15s
  scrape_interval: 30s
  scrape_timeout: 10s
  external_labels:
    prometheus_name: collector_1

rule_files:
  - /etc/prometheus/rules/*.rules

scrape_configs:
  ...

• Logs:
  On the parent instance making the remote read request:

prometheus: level=error ts=2018-07-04T06:42:50.15051076Z 
  caller=stdlib.go:89 component=web caller="http: panic serving z.z.z.z:49204" 
  msg="runtime error: invalid memory address or nil pointer dereference"

[brian-brazil]

This was fixed in 2.3.1, does that work for you?

[steve-exley]

Confirmed fixed in 2.3.1. Thanks

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.