Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an OpenTSDB datasource. #211

Merged
merged 1 commit into from Jun 14, 2014
Merged

Add an OpenTSDB datasource. #211

merged 1 commit into from Jun 14, 2014

Conversation

@mpage
Copy link

mpage commented Mar 18, 2014

This adds support for querying OpenTSDB for metric data.

Matt Page
This adds support for querying OpenTSDB for metric data.
@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Mar 18, 2014

nice! will take a look.

@torkelo torkelo added the feature label Mar 18, 2014
@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Mar 24, 2014

Sorry, might not have time to look at this as I thought. So much other stuff to do. Do you know of a good docker file to test OpenTSDB?

@mpage

This comment has been minimized.

Copy link
Author

mpage commented Mar 24, 2014

Do you know of a good docker file to test OpenTSDB?

Unfortunately I don't. A quick search results in:
https://gist.github.com/garthk/8552416

Is there anything I can do to help speed up the review process? Would screenshots of the target editor help?

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Apr 7, 2014

I am sorry this is taking long. Just have a lot on my plate and my first attempt at docker opentsdb setup failed. But I will look at it again. Yes, some screenshots of the editor would be nice :)

@pradeepchhetri

This comment has been minimized.

Copy link

pradeepchhetri commented Apr 29, 2014

Hi @mpage and @torkelo

@mpage : Thanks for adding opentsdb support.
@torkelo : If you need an opentsdb installation for testing this patch, you can use http://runabove.tsunanet.net/ . I will also test it from my side and let you know.

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Apr 29, 2014

@pradeepchhetri Thanks! Should make it easier for me to test. Felt bad for not having had time to setup opentsdb and get a feel for it so I can review and decide if this is something that should be integrated directly into Grafana. Most likely this should be fine. I just want to know what I merge into Grafana. I want to keep Grafana relatively well focused, and not be a dashboard for everything in order to keep it maintainable, polished and a great user experience. But I think the way the metric data sources work now, and metric editors, the separation is pretty good and can be better so maintaining different data sources should be fine and not create a maintainability issue in the future (hopefully).

@pradeepchhetri

This comment has been minimized.

Copy link

pradeepchhetri commented Apr 29, 2014

@torkelo Nice. I am great fan of opentsdb and one thing that opentsdb misses is one good dashboard, otherwise backend-wise its very strong. Having such a great dashboard for opentsdb will surely be widely accepted. Once you merge this code, i will surely let opentsdb community know about it.

@scicco

This comment has been minimized.

Copy link

scicco commented May 7, 2014

+1 it should be great to have this for opentsdb backends

@rtbellows

This comment has been minimized.

Copy link

rtbellows commented May 7, 2014

Tried this as well, getting a 405 error from opentsdb as grafana is sending the /query/api endpoint a request with the OPTIONS method.

@scicco

This comment has been minimized.

Copy link

scicco commented May 9, 2014

i confirm what @rtbellows told here: the request use an OPTIONS method to send query instead of GET

Is there an equivalent way to setup DEBUG=true when using opentsdb instead of graphite metric database?

@rtbellows

This comment has been minimized.

Copy link

rtbellows commented May 9, 2014

Ah, OPTIONS method is fine, but you need to configure the OpenTSDB api correctly:

http://opentsdb.net/docs/build/html/api_http/index.html#cors

by default, CORS is disabled. After enabling it, I'm getting a different error, but the prefetch using OPTIONS method is fine.

@jeffwalter

This comment has been minimized.

Copy link

jeffwalter commented May 9, 2014

OpenTSDB does not add all of the required CORS headers to allow you to run Grafana on a different host, so I cooked up the following nginx config based on what I found at http://enable-cors.org/server_nginx.html to proxy the connection and add the correct/required headers.

The following items need to be replaced by you:

  • OPENTSDB_HOST (2 instances) - Hostname or IP address of the OpenTSDB server
  • OPENTSDB_PORT (1 instance) - Port number of the OpenTSDB server
  • GRAFANA_DOMAIN (1 instance) - Domain/Hostname of the Grafana server

Here's the config:

upstream opentsdb {
  server OPENTSDB_HOST:OPENTSDB_PORT fail_timeout=0;
}

server {
  listen *:4243;

  location / {
    # Regex to whitelist systems
    if ($http_origin ~* (https?://([a-z0-9._-]*\.)?GRAFANA_DOMAIN(:[0-9]+)?)) {
      set $cors "true";
    }

    # OPTIONS indicates a CORS pre-flight request
    if ($request_method = 'OPTIONS') {
      set $cors "${cors}-options";  
    }

    # If it's OPTIONS, then it's a CORS preflight request so respond immediately with no response body
    if ($cors = "true-options") {
      add_header 'Access-Control-Allow-Origin' "$http_origin";
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
      add_header 'Content-Length' 0;
      add_header 'Content-Type' 'text/plain charset=UTF-8';
      return 204;
    }

    # Proxy the request
    proxy_set_header X-Host OPENTSDB_HOST;
    proxy_set_header X-Forwarded-For $Proxy_add_x_forwarded_for;
    proxy_set_header Authorization "";
    proxy_pass http://opentsdb;
    proxy_redirect default;
    proxy_buffer_size 16k;
    proxy_buffers 8 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_read_timeout 120;
    # Strip any OpenTSDB-generated CORS headers that overlap with our own
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Credentials';
    proxy_hide_header 'Access-Control-Allow-Headers';

    # if it's a GET or POST, set the standard CORS responses header
    if ($cors = "true") {
      # Add our own CORS headers
      add_header 'Access-Control-Allow-Origin' "$http_origin";
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' '*';
    }
  }
}

Once that bit of config is in nginx and running you need to update your
config.js to reflect the new host and/or port for OpenTSDB. It's also
possible to run the proxy on the same host as Grafana to remove the CORS
requirement, but then you'd be creating annoying dependencies... and we all
know how much ops people love annoying dependencies.

@jeffwalter

This comment has been minimized.

Copy link

jeffwalter commented May 11, 2014

Opened OpenTSDB/opentsdb#333 to add the missing Access-Control-Allow-Headers header.

@manolama

This comment has been minimized.

Copy link

manolama commented May 28, 2014

+1 Confirmed it does work with OpenTSDB. Doesn't show the tags in the key yet but it's a great start! I'd love to see this upstreamed.

@manolama

This comment has been minimized.

Copy link

manolama commented May 28, 2014

Quick caveat, doesn't look like the tags are being passed to OpenTSDB. Digging into that.

@manolama

This comment has been minimized.

Copy link

manolama commented May 28, 2014

NM, tags do work. Have to click the "+" symbol after adding it.

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented May 29, 2014

Please sign the contributor license agreement http://grafana.org/docs/contributing/cla.html

@mpage

This comment has been minimized.

Copy link
Author

mpage commented May 30, 2014

CLA signed. Please don't spam me 😛

@jeffwalter

This comment has been minimized.

Copy link

jeffwalter commented Jun 13, 2014

Any update on this pull? The OpenTSDB fix I wrote has been merged by them.

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Jun 14, 2014

That is great news. Really sorry for this taking so much time.

Will be working on grafana full time soon.

@torkelo torkelo mentioned this pull request Jun 14, 2014
@torkelo torkelo merged commit 30b62e1 into grafana:master Jun 14, 2014
1 check passed
1 check passed
default The Travis CI build passed
Details
@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Jun 14, 2014

Finally had time to setup opentsdb and try this.

Did a lot of work on polishing the editor and making it look and act good.

Seems to work. Will try to get a 1.6-rc1 out on monday.

image

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Jun 14, 2014

Did now however get CORS to work with vanilla opentsdb, the OpenTSDB/opentsdb#333 did not make a difference. OpenTSDB even with that patch (used the "next" branch) does not like the OPTIONS http method. 405 not allowed method.

Had to use nginx to proxy calls and to handle OPTIONS.

@torkelo

This comment has been minimized.

Copy link
Member

torkelo commented Jun 14, 2014

I used this nginx config & docker repo to test opentsdb https://github.com/torkelo/opentsdb-docker/blob/master/nginx.conf

@manolama

This comment has been minimized.

Copy link

manolama commented Jun 14, 2014

Thanks for the pull! To enable CORS support in opentsdb you have to set the cors domain value in the config. By default it will disable the options verb without that set. I was able to point Grafana directly at my tsds with a wildcard domain and it worked beautifully.

@jeffwalter

This comment has been minimized.

Copy link

jeffwalter commented Jun 14, 2014

I love it when a plan comes together! Thanks @manolama and @torkelo!

@mpage mpage deleted the mpage:opentsdb_datasource branch Jun 24, 2014
Dieterbe pushed a commit to Dieterbe/grafana that referenced this pull request Jul 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.