Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VictoriaMetrics(Prometheus) empty response for a query with escaping the dot by a backward slash #465

Closed
menai34 opened this issue May 10, 2020 · 1 comment
Closed

VictoriaMetrics(Prometheus) empty response for a query with escaping the dot by a backward slash #465

menai34 opened this issue May 10, 2020 · 1 comment

Comments

@menai34
Copy link

menai34 commented May 10, 2020

Hello,
We try to migrate our graphite metrics to VictoriaMetrics Cluster and have an issue with querying via carbonapi.

OS: CentOS7
carbonapi: 0.13.0-rc.1

Sent test metrics to VictoriaMetrics via Graphite plaintext protocol

#> echo "test.graphite.metric;tag1=value1;tag2=value2 321 `date +%s`" | nc -v -w3 x.x.x.x 2003
Connection to x.x.x.x port 2003 [tcp/*] succeeded!
...
#> echo "test.graphite.metric;tag1=value1;tag2=value2 213 `date +%s`" | nc -v -w3 x.x.x.x 2003
Connection to x.x.x.x port 2003 [tcp/*] succeeded!

Performed query via Grafana(Graphite protocol), query=test.graphite.*

carbonapi logs

2020-05-10T12:55:13.204+0300	DEBUG	zipper	will try to fetch data	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"]}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	single fetch	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"], "client": {}}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	waiting for slot	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "prometheus", "max_connections": 0}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	got slot	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "prometheus"}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	sending request	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "prometheus"}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	will try to fetch data	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"]}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	will do query with timeout	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "backends": [{}], "timeout": 2}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	single fetch	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "client": {}}
2020-05-10T12:55:13.204+0300	DEBUG	zipper	waiting for a slot	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "group_name": "prometheus", "backend_name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.205+0300	DEBUG	zipper	got slot	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "group_name": "prometheus", "backend_name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.205+0300	DEBUG	zipper	trying to get slot	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "find", "request": ["test.graphite.*"], "function": "HttpQuery.doRequest", "server": "http://127.0.0.1:8481/select/0/prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "uri": "http://127.0.0.1:8481/select/0/prometheus/api/v1/series?match%5B%5D=%7B__name__%3D~%22test%5C%5C.graphite%5C%5C.%5B%5E.%5D%5B%5E.%5D%2A%22%7D&start=1589104213", "name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.205+0300	DEBUG	zipper	got slot for server	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "find", "request": ["test.graphite.*"], "function": "HttpQuery.doRequest", "server": "http://127.0.0.1:8481/select/0/prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "uri": "http://127.0.0.1:8481/select/0/prometheus/api/v1/series?match%5B%5D=%7B__name__%3D~%22test%5C%5C.graphite%5C%5C.%5B%5E.%5D%5B%5E.%5D%2A%22%7D&start=1589104213", "name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	fetched response	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "group_name": "prometheus", "backend_name": "http://127.0.0.1:8481/select/0/prometheus", "response": {"Server":"http://127.0.0.1:8481/select/0/prometheus","Response":{"metrics":[{"name":"test.graphite.*","matches":[{"path":"test.graphite.metric","isLeaf":true}]}]},"Stats":{"Timeouts":0,"FindRequests":1,"FindErrors":0,"FindTimeouts":0,"RenderRequests":0,"RenderErrors":0,"RenderTimeouts":0,"InfoRequests":0,"InfoErrors":0,"InfoTimeouts":0,"SearchRequests":0,"SearchCacheHits":0,"SearchCacheMisses":0,"ZipperRequests":0,"TotalMetricsCount":0,"MemoryUsage":0,"CacheMisses":0,"CacheHits":0,"Servers":null,"FailedServers":null},"Err":null}}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	got some find responses	{"type": "broadcastGroup", "groupName": "prometheus", "type": "find", "request": ["test.graphite.*"], "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response": "&MultiGlobResponse{Metrics:[{test.graphite.* [{test.graphite.metric true}]}],}"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	single fetch	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"], "client": {}}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	waiting for slot	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "http://127.0.0.1:8481/select/0/prometheus", "max_connections": 0}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	got slot	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	sending request	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	got some target to query	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{test.graphite.metric 1589100913 1589104513 false test.graphite.* []}],}", "pathExpr": "test.graphite.*", "target": "test.graphite.metric"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	will do query	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{test.graphite.metric 1589100913 1589104513 false test.graphite.* []}],}", "query": "test\\\\.graphite\\\\.metric", "start": 1589100913, "stop": 1589104513}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	trying to get slot	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{test.graphite.metric 1589100913 1589104513 false test.graphite.* []}],}", "function": "HttpQuery.doRequest", "server": "http://127.0.0.1:8481/select/0/prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "uri": "http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range?query=test%5C%5C.graphite%5C%5C.metric&start=1589100913&step=60s&stop=1589104513", "name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.209+0300	DEBUG	zipper	got slot for server	{"type": "prometheus", "protocol": "prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{test.graphite.metric 1589100913 1589104513 false test.graphite.* []}],}", "function": "HttpQuery.doRequest", "server": "http://127.0.0.1:8481/select/0/prometheus", "name": "http://127.0.0.1:8481/select/0/prometheus", "uri": "http://127.0.0.1:8481/select/0/prometheus/api/v1/query_range?query=test%5C%5C.graphite%5C%5C.metric&start=1589100913&step=60s&stop=1589104513", "name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.214+0300	DEBUG	zipper	got response	{"type": "broadcastGroup", "groupName": "prometheus", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "http://127.0.0.1:8481/select/0/prometheus"}
2020-05-10T12:55:13.214+0300	DEBUG	zipper	got response	{"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["test.graphite.*"], "backend_name": "prometheus"}
2020-05-10T12:55:13.214+0300	DEBUG	zipper	had errors while fetching result	{"function": "FetchProtoV3", "errors": "metric not found", "errorsVerbose": "metric not found\nHTTP Code: 404\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:18\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5222\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1357\n", "httpCode": 404}
2020-05-10T12:55:13.214+0300	DEBUG	zipper	no metrics fetched	{"function": "FetchProtoV3", "errors": "metric not found", "errorsVerbose": "metric not found\nHTTP Code: 404\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:18\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5222\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.doInit\n\t/usr/lib/golang/src/runtime/proc.go:5217\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1357\n"}
2020-05-10T12:55:13.214+0300	DEBUG	render	error response or no response	{"carbonapi_uuid": "333043f5-c9d7-428c-8ea3-2d42a9183fa8", "username": "", "request_headers": {}, "error": []}
2020-05-10T12:55:13.215+0300	INFO	access	request served	{"data": {"handler":"render","carbonapi_uuid":"333043f5-c9d7-428c-8ea3-2d42a9183fa8","url":"/render","peer_ip":x.x.x.x","host":"x.x.x.x:8081","format":"json","use_cache":true,"targets":["test.graphite.*"],"cache_timeout":60,"metrics":["test.graphite.*"],"have_non_fatal_errors":true,"runtime":0.038084078,"http_code":200,"carbonapi_response_size_bytes":2,"from":1589100913,"until":1589104513,"from_raw":"-1h","until_raw":"now","uri":"/render","from_cache":false,"request_headers":{}}}

Issue related to escape of .(dot) by backward slash

#> curl -s 'http://x.x.x.x:8481/select/0/prometheus/api/v1/query_range?query=test%5C%5C.graphite%5C%5C.metric&start=1589100913&step=60s&stop=1589104513' | jq
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": []
  }
}

Query without the backward slash

#> curl -s 'http://x.x.x.x:8481/select/0/prometheus/api/v1/query_range?query=test.graphite.metric&start=1589100913&step=60s&stop=1589104513' | jq
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "test.graphite.metric",
          "tag1": "value1",
          "tag2": "value2"
        },
        "values": [
          [
            1589103360,
            "321"
          ],
          [
            1589104440,
            "213"
          ],
          [
            1589104500,
            "213"
          ],
          [
            1589104560,
            "213"
          ]
        ]
      }
    ]
  }
}

Maybe need to use regex for query {__name__=~"test\\.graphite\\.metrics"}?

curl -s 'http://x.x.x.x:8481/select/0/prometheus/api/v1/query_range?query=%7B__name__%3D~%22test%5C%5C.graphite%5C%5C.metric%22%7D&start=1589100913&step=60s&stop=1589104513' | jq
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "test.graphite.metric",
          "tag1": "value1",
          "tag2": "value2"
        },
        "values": [
          [
            1589103360,
            "321"
          ],
          [
            1589104440,
            "213"
          ],
          [
            1589104500,
            "213"
          ],
          [
            1589104560,
            "213"
          ]
        ]
      }
    ]
  }
}
@Civil Civil closed this as completed in 05e4479 May 17, 2020
@Civil
Copy link
Member

Civil commented May 17, 2020

Yup, you are right. I've fixed that in the master, it now always uses a regex query (it was already done like that for /metrics/find requests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Civil @menai34