Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"tsdb.HandleRequest() error invalid value type" #9777

Closed
faultuser opened this issue Nov 3, 2017 · 26 comments

Comments

@faultuser
Copy link

commented Nov 3, 2017

I test the Alert but throw out error:
"tsdb.HandleRequest() error invalid value type"
I use version 4.6.1
But I test same operating in version 4.5.2, no error thrown out, and working properly
image

@bergquist

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2017

What data source are you using? Could you please provide some logs?

@pathintegral

This comment has been minimized.

Copy link

commented Nov 3, 2017

Same problem here, too.
I'm using Grafana v4.6.0 with Prometheus data source.
Test notifications work with both mail and telegram.

and logs:
EROR[11-03|09:37:14] Alert Rule Result Error logger=alerting.evalHandler ruleId=0 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting

@faultuser

This comment has been minimized.

Copy link
Author

commented Nov 3, 2017

Data source is prometheus

Logs as follows:

2017/11/03 09:42:21 http: proxy error: context canceled
t=2017-11-03T09:42:21+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=172.22.117.107 time_ms=9016 size=0 referer="http://172.22.117.107:30300/dashboard/new?orgId=1&panelId=1&fullscreen&edit"
2017/11/03 09:42:23 http: proxy error: context canceled
t=2017-11-03T09:42:23+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=172.22.117.107 time_ms=2113 size=0 referer="http://172.22.117.107:30300/dashboard/new?orgId=1&panelId=1&fullscreen&edit&tab=metrics"
2017/11/03 09:42:31 http: proxy error: context canceled
t=2017-11-03T09:42:31+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=172.22.117.107 time_ms=7468 size=0 referer="http://172.22.117.107:30300/dashboard/new?orgId=1&panelId=1&fullscreen&edit&tab=metrics"
t=2017-11-03T09:43:23+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=0 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:44:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:44:01+0000 lvl=info msg="New state change" logger=alerting.resultHandler alertId=1 newState=alerting prev state=pending
t=2017-11-03T09:44:01+0000 lvl=info msg="Sending notifications for" logger=alerting.notifier ruleId=1 sent count=0
t=2017-11-03T09:44:34+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=0 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:44:40+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=0 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:45:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:45:47+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=0 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:46:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
t=2017-11-03T09:47:01+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=1 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting
@torkelo

This comment has been minimized.

Copy link
Member

commented Nov 3, 2017

Very strange, I wonder what Prometheus is returning to cause that error.

@bergquist

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2017

@f7r could you please give us the query you alert on and the result from the same query when sent from the panel. http://docs.grafana.org/installation/troubleshooting/

@dfredell

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2017

Hello,
I was just digging into this issue and found where it is. @torkelo might know the best solution to this, since he just touched it (8 days ago).

A change in #9675 is preventing me from running Test Alerts (and real alerts). I'm running Grafana v4.6.1 (commit: cac8b97) and Prometheus 2.0.0-rc.2.

UI error message:

firing:true
state:"alerting"
conditionEvals:" = true"
timeMs:"5.144ms"
error:"tsdb.HandleRequest() error invalid value type"

When the alert code gets down to

step, err := queryModel.Model.Get("step").Int64()
it checks if "step" is in the queryModel.Model and it isn't anymore because "target.step" got removed
002ac79#diff-a59431ca1f1f94cdb3ae176c50a585b2L157
Another fix could be done by setting a 'step' default, which was there in 3e73be8

@nitti

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2017

I have the same issue: Grafana 4.6.1, Prometheus data source. Log:

t=2017-11-03T14:50:06-0500 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=6196 name="Panel Title alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting

Query results are attached.
query.json.zip

@yacut

This comment has been minimized.

Copy link

commented Nov 3, 2017

Same issue here. Grafana log:

t=2017-11-03T22:12:24+0000 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=43 name="Simple alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting

Prometheus: v2.0.0-rc.2 (commit: ce63a5a8557bb33e2030a7756c58fd773736b592)
Grafana: v4.6.1 (commit: cac8b97)

@torkelo

This comment has been minimized.

Copy link
Member

commented Nov 4, 2017

Is everyone who is getting this using prometheus 2? Maybe their client lib for <2 is not compatible to query version 2

@faultuser

This comment has been minimized.

Copy link
Author

commented Nov 4, 2017

My prometheus version is 1.4.0, there is no problem with previous versions of grafana

@an-ky

This comment has been minimized.

Copy link

commented Nov 4, 2017

Get the same error with both prometheus 2.0.0-rc.2 (ce63a5a8557bb33e2030a7756c58fd773736b592) and prometheus stable 1.8.1 (3a7c51ab70fc7615cd318204d3aa7c078b7c5b20) on grafana v4.6.1 (cac8b97).

I use collectd_apache_apache_idle_workers{exported_instance="fza-web1"} as query A and the Query Inspector gives the following output

{
  "xhrStatus": "complete",
  "request": {
    "method": "GET",
    "url": "http://test-web.hs-kempten.de:9090/api/v1/query_range?query=collectd_apache_apache_idle_workers%7Bexported_instance%3D%22fza-web1%22%7D&start=1509750000&end=1509829751&step=240"
  },
  "response": {
    "status": "success",
    "data": {
      "resultType": "matrix",
      "result": [
        {
          "metric": {
            "__name__": "collectd_apache_apache_idle_workers",
            "apache": "local",
            "exported_instance": "fza-web1",
            "instance": "localhost:9091",
            "job": "prometheus"
          },
          "values": [
            [
              1509827040,
              "49"
            ],
            [
              1509827280,
              "46"
            ],
            [
              1509827520,
              "49"
            ],
            [
              1509827760,
              "49"
            ],
            [
              1509828000,
              "48"
            ],
            [
              1509828240,
              "48"
            ],
            [
              1509828480,
              "48"
            ],
            [
              1509828720,
              "49"
            ],
            [
              1509828960,
              "49"
            ],
            [
              1509829200,
              "48"
            ],
            [
              1509829440,
              "49"
            ],
            [
              1509829680,
              "49"
            ]
          ]
        }
      ]
    }
  }
}

Adding an Alert like it is shown in the screenshot below results in the mentioned error
grafik

The Grafana error log does not give any further hint (merely a replication of the output)

t=2017-11-04T22:14:01+0100 lvl=eror msg="Alert Rule Result Error" logger=alerting.evalHandler ruleId=1 name="Apache alert" error="tsdb.HandleRequest() error invalid value type" changing state to=alerting

Although I'm running Grafana in development mode and set all log levels to debug there is nothing more which shows up in the logs.

@yacut

This comment has been minimized.

Copy link

commented Nov 5, 2017

The alerts made with grafana v4.6.x have the problem. I just copied the old panel (made with grafana 4.5.1), paste the same query and the alert works...

@haizaar

This comment has been minimized.

Copy link

commented Nov 6, 2017

Same issue with Prometheus 2.0.0-rc.3 and Grafana 4.6.1.

@torkelo torkelo added type/bug and removed needs more detail labels Nov 6, 2017

@torkelo

This comment has been minimized.

Copy link
Member

commented Nov 6, 2017

We can replicate this now, happens on all new queries.

To fix this we need a proper way to handle step parameter in the backend tsdb code, the old way was very wrong as well so cannot simply revert something. We will work on a fix during the week and release a patch release early next week.

@tinder-chrisobrien

This comment has been minimized.

Copy link

commented Nov 8, 2017

Is it possible to implement as next release? Either 4.6.2 or 4.7? We are facing this issue as well.

@krdian

This comment has been minimized.

Copy link

commented Nov 8, 2017

Having same issue.

@d33ptrip

This comment has been minimized.

Copy link

commented Nov 9, 2017

Same issue here.
prometheus-1.8.0.
grafana-4.6.1

@torkelo

This comment has been minimized.

Copy link
Member

commented Nov 9, 2017

yes, expect patch release early next week

@bergquist bergquist self-assigned this Nov 13, 2017

@bergquist bergquist added this to the 4.6.2 milestone Nov 13, 2017

bergquist added a commit to bergquist/grafana that referenced this issue Nov 13, 2017

bergquist added a commit to bergquist/grafana that referenced this issue Nov 13, 2017

bergquist added a commit to bergquist/grafana that referenced this issue Nov 14, 2017

bergquist added a commit to bergquist/grafana that referenced this issue Nov 14, 2017

bergquist added a commit that referenced this issue Nov 15, 2017

prom: add support for default step param (#9866)
Alerting for prometheus have been depending on the step parameter from each query.
In #9226 we changed the behavior for step in the
frontend which caused problems for alerting. This commit fixes that by introducing a default
min interval value so alerting always have something to depend on. 

closes #9777

bergquist added a commit that referenced this issue Nov 15, 2017

prom: add support for default step param (#9866)
Alerting for prometheus have been depending on the step parameter from each query.
In #9226 we changed the behavior for step in the
frontend which caused problems for alerting. This commit fixes that by introducing a default
min interval value so alerting always have something to depend on.

closes #9777
(cherry picked from commit 5d6ed6c)

bergquist added a commit that referenced this issue Nov 15, 2017

@bergquist

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2017

Alerting for Prometheus have been depending on the step parameter defined in the frontend query.
In #9226 we changed the behavior for step in the
frontend which caused problems for alerting. The old solution was also problematic if you had a long time range in the dashboard but short time range in the alert query since the step parameter was reused.

So instead of reverting back to the old behavior, we decided to make a proper fix for this problem. We introduced a new setting for the Prometheus data source to allow you to set a min step parameter for all queries for that data source. This value should align with the scrape interval in your Prometheus server. By default its set to 15s

This fix will be included in the 4.6.2 release. You can also get it by running the latest master release.

ryantxu added a commit to NatelEnergy/grafana that referenced this issue Nov 15, 2017

Merge remote-tracking branch 'grafana/master'
* grafana/master: (62 commits)
  changelog: make prom fixes more explicit
  changelog: adds note about closing grafana#9777
  prom: add support for default step param (grafana#9866)
  properly escape components of connection string (grafana#9850)
  refactor: changed string slicing to strings.TrimPrefix, grafana#9862
  build: fixed jshint error
  sync documentation, add remark about to_timestamp and redshift (grafana#9841)
  Update CHANGELOG.md
  fix: Html escaping caused issue in InfluxDB query editor,  could not pick greater than or less then operators, fixes grafana#9871
  changelog: adds note about closing grafana#8523
  teams: removes print statement
  Add Microsoft Teams notifier
  docs: update building from source doc with node-gyp
  Update CHANGELOG.md
  heatmap: fix tooltip in "Time series bucket" mode, grafana#9332 (grafana#9867)
  fix: Table panel now renders annotations correctly. Fixes grafana#9842 (grafana#9868)
  build: fixes build and jest tests on Windows
  Update CHANGELOG.md
  fix cloudwatch ec2_instance_attribute (grafana#9718)
  graph: the stack & legend sort sync was not working correctly, the z-index sorting that happened in after the legend sort order was applied and messed with the order even though the sort function returned zero for all entries, combined the sort function to one sort function, fixes grafana#9789 (grafana#9797)
  ...
@dfredell

This comment has been minimized.

Copy link
Contributor

commented Nov 17, 2017

Thanks,
I just upgraded to docker Grafana 4.6.2 and my Prometheus "Test Rule" on alerts works again.
Prometheus 2.0.0

@haizaar

This comment has been minimized.

Copy link

commented Nov 17, 2017

@cbrake

This comment has been minimized.

Copy link

commented Nov 17, 2017

working here -- many thanks for this fix!

@littleiffel

This comment has been minimized.

Copy link

commented Nov 17, 2017

Thanks for this quick fix. I confirm it is also working with Prometheus v2.0.0

@carlbennett

This comment has been minimized.

Copy link

commented Nov 17, 2017

Confirmed. I was looking for the cause and stumbled upon this issue. After updating from Grafana 4.6.1 to 4.6.2, the problem has been resolved. Thanks for the fix!

P.S. I am using Prometheus 1.8.2 and node_exporter 0.15.1.

@pathintegral

This comment has been minimized.

Copy link

commented Nov 18, 2017

working here too!
thanks guys.

@arunasai

This comment has been minimized.

Copy link

commented Jun 20, 2018

Is the fix present v.5.0.0 beta version of Grafana

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.