500 Internal Server Error when query metrics using label __name__

[tuxknight]

Bug Report

What did you do?
query metrics using promQL

one simple query request
{__name__=~"node_exporter_build_info",branch!="HEAD"}

one complex query request
irate({__name__=~"node_network_receive_bytes(_total|)",instance="1.2.3.4:9100",job="nodes",device!~"^(lo|veth.*|tunl.*|cali.*|docker.*)"}[1m])

What did you expect to see?

prometheus returns http code 200 .

What did you see instead? Under which circumstances?

HTTP status 500 Internal Server Error
{"status":"error","errorType":"internal","error":"server returned HTTP status 500 Internal Server Error"}
Environment

• System information:

  Linux 3.10.0-327.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.5.0 (branch: HEAD, revision: 67dc912ac8b24f94a1fc478f352d25179c94ab9b) build user: root@578ab108d0b9 build date: 20181106-11:40:44 go version: go1.11.1

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 30s 
  scrape_timeout:      10s

  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'master'

remote_read:
  - url: http://101.21.36.42:8086/api/v1/prom/read?db=prometheus
    read_recent: true
remote_write:
  - url: http://101.21.36.42:8086/api/v1/prom/write?db=prometheus

rule_files:
  - /data/application/prometheus/rules/*.rules

# Alertmanager configuration
alerting:
  alertmanagers:
    - scheme: http
      path_prefix: /am
      static_configs:
        - targets:
                - 101.21.36.42::9093

# A list of scrape configurations.
scrape_configs:
  - job_name: 'prometheus'
    metrics_path: "/prome/metrics"
    scrape_interval: 10s
    scrape_timeout:  10s
    static_configs:
      - targets:
        - 101.21.36.42:9090

  - job_name: 'push_gateway'
    static_configs:
    - targets:
        - 101.21.36.42:9091

  - job_name: 'node'
    file_sd_configs:
      - files:
        - /data/application/prometheus/file_ds/nodes.json
    relabel_configs:
    - source_labels: [__address__]
      regex: (.+):(.+)
      replacement: ${1}
      target_label: instance_ip


  - job_name: 'snmp'
    scrape_interval: 120s
    scrape_timeout: 120s
    file_sd_configs:
      - files:
        - /data/application/prometheus/file_ds/snmps.json
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__address__]
        target_label: instance_ip
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 101.21.36.42:9116  

• Logs:

Dec 25 15:38:04 localhost prometheus: level=error ts=2018-12-25T07:38:04.93170476Z caller=engine.go:526 component="query engine" msg="error selecting series set" err="server returned HTTP status 500 Internal Server Error"
Dec 25 15:38:05 localhost prometheus: level=error ts=2018-12-25T07:38:05.275255324Z caller=engine.go:526 component="query engine" msg="error selecting series set" err="server returned HTTP status 500 Internal Server Error"

[roidelapluie]

Maybe you are hitting:

#4562
#4589

irate({__name__=~"node_network_receive_bytes(_total|)",instance="1.2.3.4:9100",job="nodes",device!~"^(lo|veth.*|tunl.*|cali.*|docker.*)"}[1m])

should be

irate(node_network_receive_bytes{instance="1.2.3.4:9100",job="nodes",device!~"^(lo|veth.*|tunl.*|cali.*|docker.*)"}[1m]) or irate(node_network_receive_bytes_total{instance="1.2.3.4:9100",job="nodes",device!~"^(lo|veth.*|tunl.*|cali.*|docker.*)"}[1m])

[codesome]

In #4562 you seem to get no data (or something like that), but did you also see 500 error with that @roidelapluie ?

[roidelapluie]

No but since that pr you should get a 400 or 500 (need to check that later)

[simonpasquier]

You're using remote read (with InfluxDB?) and the error bubbles up from the remote system. Check the logs there.