Add an OpenTSDB datasource.

[mpage]

This adds support for querying OpenTSDB for metric data.

[torkelo]

nice! will take a look.

[torkelo]

Sorry, might not have time to look at this as I thought. So much other stuff to do. Do you know of a good docker file to test OpenTSDB?

[mpage]

Do you know of a good docker file to test OpenTSDB?

Unfortunately I don't. A quick search results in:
https://gist.github.com/garthk/8552416

Is there anything I can do to help speed up the review process? Would screenshots of the target editor help?

[torkelo]

I am sorry this is taking long. Just have a lot on my plate and my first attempt at docker opentsdb setup failed. But I will look at it again. Yes, some screenshots of the editor would be nice :)

[pradeepchhetri]

Hi @mpage and @torkelo

@mpage : Thanks for adding opentsdb support.
@torkelo : If you need an opentsdb installation for testing this patch, you can use http://runabove.tsunanet.net/ . I will also test it from my side and let you know.

[torkelo]

@pradeepchhetri Thanks! Should make it easier for me to test. Felt bad for not having had time to setup opentsdb and get a feel for it so I can review and decide if this is something that should be integrated directly into Grafana. Most likely this should be fine. I just want to know what I merge into Grafana. I want to keep Grafana relatively well focused, and not be a dashboard for everything in order to keep it maintainable, polished and a great user experience. But I think the way the metric data sources work now, and metric editors, the separation is pretty good and can be better so maintaining different data sources should be fine and not create a maintainability issue in the future (hopefully).

[pradeepchhetri]

@torkelo Nice. I am great fan of opentsdb and one thing that opentsdb misses is one good dashboard, otherwise backend-wise its very strong. Having such a great dashboard for opentsdb will surely be widely accepted. Once you merge this code, i will surely let opentsdb community know about it.

[scicco]

+1 it should be great to have this for opentsdb backends

[rtbellows]

Tried this as well, getting a 405 error from opentsdb as grafana is sending the /query/api endpoint a request with the OPTIONS method.

[scicco]

i confirm what @rtbellows told here: the request use an OPTIONS method to send query instead of GET

Is there an equivalent way to setup DEBUG=true when using opentsdb instead of graphite metric database?

[rtbellows]

Ah, OPTIONS method is fine, but you need to configure the OpenTSDB api correctly:

http://opentsdb.net/docs/build/html/api_http/index.html#cors

by default, CORS is disabled. After enabling it, I'm getting a different error, but the prefetch using OPTIONS method is fine.

[itsjohannawren]

OpenTSDB does not add all of the required CORS headers to allow you to run Grafana on a different host, so I cooked up the following nginx config based on what I found at http://enable-cors.org/server_nginx.html to proxy the connection and add the correct/required headers.

The following items need to be replaced by you:

• OPENTSDB_HOST (2 instances) - Hostname or IP address of the OpenTSDB server
• OPENTSDB_PORT (1 instance) - Port number of the OpenTSDB server
• GRAFANA_DOMAIN (1 instance) - Domain/Hostname of the Grafana server

Here's the config:

upstream opentsdb {
  server OPENTSDB_HOST:OPENTSDB_PORT fail_timeout=0;
}

server {
  listen *:4243;

  location / {
    # Regex to whitelist systems
    if ($http_origin ~* (https?://([a-z0-9._-]*\.)?GRAFANA_DOMAIN(:[0-9]+)?)) {
      set $cors "true";
    }

    # OPTIONS indicates a CORS pre-flight request
    if ($request_method = 'OPTIONS') {
      set $cors "${cors}-options";  
    }

    # If it's OPTIONS, then it's a CORS preflight request so respond immediately with no response body
    if ($cors = "true-options") {
      add_header 'Access-Control-Allow-Origin' "$http_origin";
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
      add_header 'Content-Length' 0;
      add_header 'Content-Type' 'text/plain charset=UTF-8';
      return 204;
    }

    # Proxy the request
    proxy_set_header X-Host OPENTSDB_HOST;
    proxy_set_header X-Forwarded-For $Proxy_add_x_forwarded_for;
    proxy_set_header Authorization "";
    proxy_pass http://opentsdb;
    proxy_redirect default;
    proxy_buffer_size 16k;
    proxy_buffers 8 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_read_timeout 120;
    # Strip any OpenTSDB-generated CORS headers that overlap with our own
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Credentials';
    proxy_hide_header 'Access-Control-Allow-Headers';

    # if it's a GET or POST, set the standard CORS responses header
    if ($cors = "true") {
      # Add our own CORS headers
      add_header 'Access-Control-Allow-Origin' "$http_origin";
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' '*';
    }
  }
}

Once that bit of config is in nginx and running you need to update your
config.js to reflect the new host and/or port for OpenTSDB. It's also
possible to run the proxy on the same host as Grafana to remove the CORS
requirement, but then you'd be creating annoying dependencies... and we all
know how much ops people love annoying dependencies.

[itsjohannawren]

Opened OpenTSDB/opentsdb#333 to add the missing Access-Control-Allow-Headers header.

[manolama]

+1 Confirmed it does work with OpenTSDB. Doesn't show the tags in the key yet but it's a great start! I'd love to see this upstreamed.

[manolama]

Quick caveat, doesn't look like the tags are being passed to OpenTSDB. Digging into that.

[manolama]

NM, tags do work. Have to click the "+" symbol after adding it.

[torkelo]

Please sign the contributor license agreement http://grafana.org/docs/contributing/cla.html

[mpage]

CLA signed. Please don't spam me 😛

[itsjohannawren]

Any update on this pull? The OpenTSDB fix I wrote has been merged by them.

[torkelo]

That is great news. Really sorry for this taking so much time.

Will be working on grafana full time soon.

[torkelo]

Finally had time to setup opentsdb and try this.

Did a lot of work on polishing the editor and making it look and act good.

Seems to work. Will try to get a 1.6-rc1 out on monday.

[torkelo]

Did now however get CORS to work with vanilla opentsdb, the OpenTSDB/opentsdb#333 did not make a difference. OpenTSDB even with that patch (used the "next" branch) does not like the OPTIONS http method. 405 not allowed method.

Had to use nginx to proxy calls and to handle OPTIONS.

[torkelo]

I used this nginx config & docker repo to test opentsdb https://github.com/torkelo/opentsdb-docker/blob/master/nginx.conf

[manolama]

Thanks for the pull! To enable CORS support in opentsdb you have to set the cors domain value in the config. By default it will disable the options verb without that set. I was able to point Grafana directly at my tsds with a wildcard domain and it worked beautifully.

[itsjohannawren]

I love it when a plan comes together! Thanks @manolama and @torkelo!