Need some FQA while using grafana-spark-dashboards

[tsingfu]

The show in README.md is cool! Good job.
I did a test using grafana-spark-dashboards with my graphite, and I got some errors on web:

Script Error
Please make sure it exists and returns a valid dashboard

the detailed error message could be found in spark-user-list Monitoring Spark with Graphite and Grafana.
I could not find any document about it, so I need some help.
Thanks.

[ryan-williams]

Hi @tsingfu, thanks for the detailed report on the email list (seems like it hasn't been accepted by the list yet for some reason?).

I think I see the problem: the /ws/v1/cluster/apps XHR is supposed to hit your YARN RM's JSON API and pull informations about what YARN apps exist, but it is 404'ing:

That request happens here (note that I just pushed an update that makes this request default to localhost:8091 instead of a hardcoded domain that I had in there previously that was specific to my cluster).

It seems that you've already found and changed that request, to http://monitor1:61081/ws/v1/cluster/apps, but your screenshot of your YARN RM web UI lives at http://spark1:8088/....

So, you actually want to have spark.js try to hit http://spark1:8088/ws/v1/cluster/apps, which should be your YARN RM's JSON API (try this with a curl first to be sure).

However, you probably don't have CORS headers set to allow your Grafana (monitor1:61081) to make XHRs to your YARN RM (spark1:8088). The work-around to this that I've described in the README is to run a proxy that hits the YARN JSON API, adds the appropriate CORS headers, and returns the info to Grafana. See this section of my example nginx.conf. You'll want to change the demeter-mgmt1.demeter.hpc.mssm.edu to spark1, I think. That will then listen on the same machine as Grafana is running on, on port 8091, so the localhost:8091 XHR will correctly return YARN app JSON to you.

Sorry this is a lot of explanation, let me know if it doesn't make sense and I can try to explain it differently / more directly.

[tsingfu]

Hi, @ryan-williams . I am very nice to hear from you.
I find I have troubles to try the work-around for CORS problem.

• First , I am not sure how to configure proxy in my apcahe config,may we add example for apache?
• So I installed nginx and configured grafana according your example nginx.conf.

My configuration in nginx.conf:

  # configure for grafana using 62081
  server {
    listen 62081 default_server;
    server_name monitor1;

    root /data/grafana/;

    location / {
      index index.html;
    }
  }

  # configure for grafana-spark-dashboard, Use port 62088 as a proxy to YARN RM JSON API.
  server {
    listen 62088;
    server_name monitor1;
    add_header Access-Control-Allow-Origin "monitor1";
    location / {
      proxy_pass http://spark1:8088;
    }
  }

• Result:
  when accessing http://monitor1:62081/#/dashboard/script/spark.js?app=application_1426815734479_0001
  I got the same error on web:

Script Error
Please make sure it exists and returns a valid dashboard

the detailed errors in web console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://spark1:8088/ws/v1/cluster/apps. This can be fixed by moving the resource to the same domain or enabling CORS.
Error:  app.496fdc2e.js line 22 > Function:43:12
Object { readyState: 0, getResponseHeader: .ajax/v.getResponseHeader(), getAllResponseHeaders: .ajax/v.getAllResponseHeaders(), setRequestHeader: .ajax/v.setRequestHeader(), overrideMimeType: .ajax/v.overrideMimeType(), statusCode: .ajax/v.statusCode(), abort: .ajax/v.abort(), state: .Deferred/d.state(), always: .Deferred/d.always(), then: .Deferred/d.then(), 11 more… } "error" "" app.496fdc2e.js line 22 > Function:42:6

"Error: apps is null
findYarnApp@http://monitor1:62081/app/app.496fdc2e.js line 22 > Function:52:3
getYarnAppInfo@http://monitor1:62081/app/app.496fdc2e.js line 22 > Function:100:15
anonymous@http://monitor1:62081/app/app.496fdc2e.js line 22 > Function:121:11
m@http://monitor1:62081/app/app.496fdc2e.js:22:30851
i@http://monitor1:62081/app/app.496fdc2e.js:9:30874
j/<@http://monitor1:62081/app/app.496fdc2e.js:9:31046
Rc/this.$get</l.prototype.$eval@http://monitor1:62081/app/app.496fdc2e.js:10:6192
Rc/this.$get</l.prototype.$digest@http://monitor1:62081/app/app.496fdc2e.js:10:4706
Rc/this.$get</l.prototype.$apply@http://monitor1:62081/app/app.496fdc2e.js:10:6450
h@http://monitor1:62081/app/app.496fdc2e.js:9:14680
q@http://monitor1:62081/app/app.496fdc2e.js:9:16489
kc/</u.onload@http://monitor1:62081/app/app.496fdc2e.js:9:17029
" app.496fdc2e.js:9:25524

"Script dashboard error TypeError: apps is null"

Any help is highly appreciated.

[ryan-williams]

It seems like your setup is:

• monitor1:62081: grafana server
• monitor1:62088: proxy to spark1:8088
• spark1:8088: YARN RM

That all seems reasonable. However, the first error line you've pasted:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://spark1:8088/ws/v1/cluster/apps. This can be fixed by moving the resource to the same domain or enabling CORS.

implies that you are making an XHR from Grafana (monitor1:62081) directly to YARN (spark1:8088).

You need to tell Grafana to fetch YARN info from your proxy (monitor1:62088). If Grafana (monitor1:62081) tries to make an XHR to spark1:8088, you will get Cross-Origin Request Blocked, because spark1:8088 doesn't add a CORS header whitelisting monitor1:62081; your proxy does add that header, and that's the whole point of having it.

In spark.js (here), you need to tell your dashboard to fetch YARN info from your proxy (monitor1:62088) instead of from your YARN RM directly (spark1:8088).

Note that if we could tell YARN RM (spark1:8088) to add its own CORS headers to responses, we wouldn't need the proxy; from my last reading of YARN docs, it seemed like that might be coming in a future/next version of YARN, but does not exist on the version I am running, which is why I went this proxy route.

re: configuring Apache to do this, I always get confused trying to configure apache web server, but a line like this:

             Header set Access-Control-Allow-Origin "http://monitor1:62081"

inside e.g. a vhost for monitor1:62088 should add the correct headers. Here's some docs.

I've found it useful to sanity-check these things by running e.g.:

$ curl -I http://monitor1:62088/ws/v1/cluster/apps

curl -I will print only the response headers. You should see a CORS line in the headers that are printed. If you curl without -I, you should see a JSON blob that your proxy has fetched from YARN RM itself (spark1:8088).

For my setup, I have grafana at demeter-login2.demeter.hpc.mssm.edu:8090 and my proxy at demeter-login2.demeter.hpc.mssm.edu:8091, per my example nginx config, and I can see this:

$ curl -I http://demeter-login2.demeter.hpc.mssm.edu:8091/ws/v1/cluster/apps
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 20 Mar 2015 17:09:24 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
Cache-Control: no-cache
Expires: Fri, 20 Mar 2015 17:09:24 GMT
Pragma: no-cache
Expires: Fri, 20 Mar 2015 17:09:24 GMT
Pragma: no-cache
Access-Control-Allow-Origin: http://demeter-login2.demeter.hpc.mssm.edu:8090

So I am able to request /ws/v1/cluster/apps from my proxy (demeter-login2.…:8091), it gets the data from my YARN RM (http://demeter-mgmt1.demeter.hpc.mssm.edu:8088, per this line in nginx.conf), and adds the CORS header (last line of the code block above) that allows that info to be returned to Grafana, which is running at demeter-login2.…:8090.

I know this is confusing, let me know if it still doesn't make sense.

[tsingfu]

@ryan-williams , It works now. Thanks very much for your help.

I think support of monitoring standalone mode could be in todo list.

[ryan-williams]

great, glad to hear it @tsingfu, I've filed #6 as a place to capture work related to making the Spark-standalone story easier

[ryan-williams]

I'm going to close this out, let me know / reopen if there are things you think we should e.g. add to the documentation