'Admin View' of save object relationships

[LeeDr]

Describe the feature: I've had a few occasions where it would have been very useful to be able to quickly see the hierarchy of saved objects in a dashboard. For example, I open a dashboard while logged in as a user who has access to logstash-* but not metricbeat-* and, as expected, I get an error about the metricbeat data. But which visualization in the dashboard is using metricbeat data? The visualization name doesn't always give it away.

What I'd like to see is something like this;

• First we have Dashboards on the left
• the next column lists the Visualizations on those Dashboards
• some Visualizations are not on dashboards so they're at the end without a dashboard
• the next column are Searches, either the ones used by the visualizations or one directly on Dashboards
• there might also be some Searches not used in any Dashboards or Visualizations so they would be listed at the end
• the last column are Indexes (index patterns actually) and would be the source of the Visualization and/or Search

Everything in the table would be a link to jump directly to that view.

It might make sense to support sorting on the columns to make it easy to find, for example, that some visualization is used on 3 different dashboards.

It might also be useful to allow filtering at least on some columns. Maybe just on Dashbaords?

Dashboard	Visualization	Search	Index
Packetbeat Cassandra	Cassandra ResponseKeyspace	Cassandra search	packetbeat-*
Packetbeat Cassandra	Cassandra ResponseType	Cassandra search	packetbeat-*
Packetbeat Cassandra	Cassandra ResponseTime	Cassandra search	packetbeat-*
Packetbeat Cassandra	Cassandra RequestCount	Cassandra search	packetbeat-*
Packetbeat Cassandra	Cassandra Ops		packetbeat-*
Packetbeat Cassandra	Cassandra RequestCountStackByType		packetbeat-*
Packetbeat Cassandra	Cassandra ResponseCountStackByType		packetbeat-*
Packetbeat Cassandra	Cassandra RequestCountByType		packetbeat-*
Packetbeat Cassandra	Cassandra ResponseCountByType		packetbeat-*
Packetbeat Cassandra	Navigation
Packetbeat Cassandra		Cassandra QueryView	packetbeat-*
Packetbeat Thrift performance	Navigation		packetbeat-*
Packetbeat Thrift performance	Thrift requests per minute		packetbeat-*
Packetbeat Thrift performance	Thrift RPC Errors		packetbeat-*
Packetbeat Thrift performance	Slowest Thrift RPC methods		packetbeat-*
Packetbeat Thrift performance	Thrift response times percentiles		packetbeat-*
	Top Thrift RPC methods		packetbeat-*
	Top Thrift RPC calls with errors		packetbeat-*

I started a little experiment in the selenium test framework just because I'm getting familiar with the elasticsearch client there. My first attempt started with getting the dashboards and parsing out the panels (which could be visualizations or saved searches) and listing them out.

    return esClient.search('.kibana', 'dashboard')
    .then((response) => {
      PageObjects.common.debug('found ' + response.hits.total + ' dashboards');

I parsed the visualization ids out of each dashboard object like this;

JSON.parse(response.hits.hits._source.panelsJSON).id

And then I tried doing a GET on each visualization.id. This was sort of working, but I wasn't getting the console output how I wanted and probably needs to be re-thought if it were to end up in Kibana or as a plugin.

I chatted with @cjcenizal about it and agreed it would be much more efficient to just do 3 searches, 1 for Dashboards, 1 for Visualizations, and 1 for Searches, and join those results afterwards.

[cjcenizal]

I think we should also explore alternative ways to visualize this data. For example, how about a parallel coordinates chart?

Interactive examples:

• http://mbostock.github.io/d3/talk/20111116/iris-parallel.html
• http://mbostock.github.io/d3/talk/20111018/tree.html
• http://bl.ocks.org/robschmuecker/7880033
• http://vizuly.io/product/weighted-tree/?demo=d3js
• http://www.brightpointinc.com/2015-nfl-predictions/
• http://csclub.uwaterloo.ca/~n2iskand/?page_id=13

Another option is a Sankey diagram:

Interactive examples:

• https://bost.ocks.org/mike/sankey/

[LeeDr]

That looks pretty cool, but I'm not sure how well it would scale to a large number of visualizations. If it were interactive and mousing over a object would highlight it's line and dim the others like our other charts do that might help a lot. And/or if you could apply filtering on some columns like Dashboards so that you're only looking at a subset of the data it might be great.

[LeeDr]

I have a script that generates this table below (except the header).
Picture these each as a link to that Dashboard, Visualization, Saved Search, or Index pattern.
The code does a search on .kibana for index-pattern, search, visualization, and dashboard and stores the result in a hashmap with the object id as the key.
Then it iterates through dashboards, and through the panels of the dashboard.

  If the panel is a visualization
     if the visualization came from a saved search
        get the saved search, 
           get the index pattern
     if the visualization used a "new query" on the index pattern
          get the index pattern
     else (it was a markdown visualization)
  If the panel is a search
          get the index pattern

What's not included here, but probably should be listed, are Visualization and Saved Searches which are not used on any Dashboards. They should be listed at the end. I guess I'd have to set a flag on each Visualization and Saved Search object in the hashmap as it's used to know which ones aren't used. Same for Index patterns.

Dashboard	Visualization	Saved Search	Index Pattern
Metricbeat-filesystem	System Navigation
Metricbeat-filesystem	Top hosts by disk size	Fsstats	metricbeat-*
Metricbeat-filesystem	Disk space overview	Fsstats	metricbeat-*
Metricbeat-filesystem	Free disk space over days	Fsstats	metricbeat-*
Metricbeat-filesystem	Total files over days	Fsstats	metricbeat-*
Metricbeat-processes	Top processes by memory usage	Process stats	metricbeat-*
Metricbeat-processes	CPU usage per process	Process stats	metricbeat-*
Metricbeat-processes	Memory usage per process	Process stats	metricbeat-*
Metricbeat-processes	Top processes by CPU usage	Process stats	metricbeat-*
Metricbeat-processes	System Navigation
Metricbeat-processes	Number of Pids	Process stats	metricbeat-*
Metricbeat-processes	Number of processes	Process stats	metricbeat-*
Packetbeat Cassandra	Cassandra: ResponseKeyspace		packetbeat-*
Packetbeat Cassandra	Cassandra: ResponseType		packetbeat-*
Packetbeat Cassandra	Cassandra: ResponseTime		packetbeat-*
Packetbeat Cassandra	Cassandra: RequestCount		packetbeat-*
Packetbeat Cassandra	Cassandra: Ops		packetbeat-*
Packetbeat Cassandra	Cassandra: RequestCountStackByType		packetbeat-*
Packetbeat Cassandra	Cassandra: ResponseCountStackByType		packetbeat-*
Packetbeat Cassandra	Cassandra: RequestCountByType		packetbeat-*
Packetbeat Cassandra	Cassandra: ResponseCountByType		packetbeat-*
Packetbeat Cassandra	Navigation
Packetbeat Cassandra		Cassandra: QueryView	packetbeat-*
Packetbeat Thrift performance	Navigation
Packetbeat Thrift performance	Thrift requests per minute	Thrift transactions	packetbeat-*
Packetbeat Thrift performance	Thrift RPC Errors	Thrift errors	packetbeat-*
Packetbeat Thrift performance	Slowest Thrift RPC methods	Thrift transactions	packetbeat-*
Packetbeat Thrift performance	Thrift response times percentiles	Thrift transactions	packetbeat-*
Packetbeat Thrift performance	Top Thrift-RPC methods	Thrift transactions	packetbeat-*
Packetbeat Thrift performance	Top Thrift-RPC calls with errors	Thrift errors	packetbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - CPU	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Hostname list	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Load1/5/15	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Scoreboard	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Total accesses and kbytes	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Uptime	Apache HTTPD	metricbeat-*
Metricbeat - Apache HTTPD server status	Apache HTTPD - Workers	Apache HTTPD	metricbeat-*
Metricbeat-cpu	System Navigation
Metricbeat-cpu	CPU usage over time	Cpu stats	metricbeat-*
Metricbeat-cpu	System Load	Load stats	metricbeat-*
Metricbeat-cpu	System Load over time	Load stats	metricbeat-*
Metricbeat-cpu	Top hosts by CPU usage	Cpu-Load stats	metricbeat-*
Metricbeat-cpu	CPU Usage	Cpu stats	metricbeat-*
Packetbeat HTTP	Web transactions	Web transactions	packetbeat-*
Packetbeat HTTP	HTTP error codes		packetbeat-*
Packetbeat HTTP	HTTP error codes evolution		packetbeat-*
Packetbeat HTTP	Navigation
Packetbeat HTTP	Total number of HTTP transactions	Web transactions	packetbeat-*
Packetbeat HTTP	HTTP codes for the top queries	Web transactions	packetbeat-*
Packetbeat HTTP	Top 10 HTTP requests	Web transactions	packetbeat-*
Winlogbeat Dashboard	Number of Events Over Time By Event Log		winlogbeat-*
Winlogbeat Dashboard	Number of Events		winlogbeat-*
Winlogbeat Dashboard	Top Event IDs		winlogbeat-*
Winlogbeat Dashboard	Event Levels		winlogbeat-*
Winlogbeat Dashboard	Event Sources		winlogbeat-*
Metricbeat-memory	System Navigation
Metricbeat-memory	Top hosts by memory usage	Memory stats	metricbeat-*
Metricbeat-memory	Memory usage over time	Memory stats	metricbeat-*
Metricbeat-memory	Swap usage over time	Memory stats	metricbeat-*
Metricbeat-memory	Total Memory	Memory stats	metricbeat-*
Metricbeat-memory	Available Memory	Memory stats	metricbeat-*
Metricbeat-memory	Memory usage	Memory stats	metricbeat-*
Metricbeat-memory	Swap usage	Memory stats	metricbeat-*
Metricbeat-overview	Servers overview	System stats	metricbeat-*
Metricbeat-overview	System Navigation
Packetbeat Dashboard	Web transactions	Web transactions	packetbeat-*
Packetbeat Dashboard	DB transactions	DB transactions	packetbeat-*
Packetbeat Dashboard	Cache transactions	Cache transactions	packetbeat-*
Packetbeat Dashboard	RPC transactions	RPC transactions	packetbeat-*
Packetbeat Dashboard	Response times percentiles	Packetbeat Search	packetbeat-*
Packetbeat Dashboard	Errors count over time	Transactions errors	packetbeat-*
Packetbeat Dashboard	Errors vs successful transactions	Packetbeat Search	packetbeat-*
Packetbeat Dashboard	Latency histogram	Packetbeat Search	packetbeat-*
Packetbeat Dashboard	Client locations	Packetbeat Search	packetbeat-*
Packetbeat Dashboard	Response times repartition	Packetbeat Search	packetbeat-*
Packetbeat Dashboard	Navigation
Packetbeat MongoDB performance	Navigation
Packetbeat MongoDB performance	MongoDB errors	MongoDB errors	packetbeat-*
Packetbeat MongoDB performance	MongoDB commands	MongoDB transactions	packetbeat-*
Packetbeat MongoDB performance	MongoDB errors per collection	MongoDB errors	packetbeat-*
Packetbeat MongoDB performance	MongoDB in/out throughput	MongoDB transactions	packetbeat-*
Packetbeat MongoDB performance	MongoDB response times by collection	MongoDB transactions	packetbeat-*
Packetbeat MongoDB performance	Top slowest MongoDB queries	MongoDB transactions	packetbeat-*
Packetbeat MongoDB performance	Number of MongoDB transactions with writeConcern w=0	MongoDB transactions with write concern 0	packetbeat-*
Packetbeat MySQL performance	MySQL Errors	MySQL errors	packetbeat-*
Packetbeat MySQL performance	MySQL Methods	MySQL Transactions	packetbeat-*
Packetbeat MySQL performance	Navigation
Packetbeat MySQL performance	MySQL throughput	MySQL Transactions	packetbeat-*
Packetbeat MySQL performance	Most frequent MySQL queries	MySQL Transactions	packetbeat-*
Packetbeat MySQL performance	Slowest MySQL queries	MySQL Transactions	packetbeat-*
Packetbeat MySQL performance	Mysql response times percentiles	MySQL Transactions	packetbeat-*
Packetbeat MySQL performance	MySQL Reads vs Writes	MySQL Transactions	packetbeat-*
Packetbeat NFS	NFS clients pie chart	nfs	packetbeat-*
Packetbeat NFS	NFS operations area chart	nfs	packetbeat-*
Packetbeat NFS	NFS top group pie chart	nfs	packetbeat-*
Packetbeat NFS	NFS top users pie chart	nfs	packetbeat-*
Packetbeat NFS	NFS response times	nfs	packetbeat-*
Packetbeat NFS	NFS errors	nfs	packetbeat-*
Packetbeat NFS	NFS operation table	nfs	packetbeat-*
Packetbeat NFS	NFS bytes in / out	nfs	packetbeat-*
Packetbeat NFS	Navigation
Metricbeat system overview	Network Bytes	Network data	metricbeat-*
Metricbeat system overview	Network Packetloss	Network data	metricbeat-*
Metricbeat system overview	System Navigation
Metricbeat system overview	Total Memory	Memory stats	metricbeat-*
Metricbeat system overview	Available Memory	Memory stats	metricbeat-*
Metricbeat system overview	System overview by host	System stats	metricbeat-*
Metricbeat system overview	System Load	Load stats	metricbeat-*
Metricbeat system overview	CPU Usage	Cpu stats	metricbeat-*
Metricbeat filesystem per Host	Top disks by memory usage	Filesystem stats	metricbeat-*
Metricbeat filesystem per Host	Disk utilization over time	Filesystem stats	metricbeat-*
Metricbeat filesystem per Host	System Navigation
Metricbeat filesystem per Host	Disk space distribution	Filesystem stats	metricbeat-*
Metricbeat-network	In vs Out Network Bytes	Network data	metricbeat-*
Metricbeat-network	Top 10 interfaces	Network data	metricbeat-*
Metricbeat-network	Network Packetloss	Network data	metricbeat-*
Metricbeat-network	Packet loss on interfaces	Network data	metricbeat-*
Metricbeat-network	System Navigation
Metricbeat-network	Network Bytes	Network data	metricbeat-*
Packetbeat PgSQL performance	Navigation
Packetbeat PgSQL performance	PgSQL Errors	PgSQL errors	packetbeat-*
Packetbeat PgSQL performance	PgSQL Methods	PgSQL transactions	packetbeat-*
Packetbeat PgSQL performance	PgSQL response times percentiles	PgSQL transactions	packetbeat-*
Packetbeat PgSQL performance	PgSQL throughput	PgSQL transactions	packetbeat-*
Packetbeat PgSQL performance	PgSQL Reads vs Writes	PgSQL transactions	packetbeat-*
Packetbeat PgSQL performance	Most frequent PgSQL queries	PgSQL transactions	packetbeat-*
Packetbeat PgSQL performance	Slowest PgSQL queries	PgSQL transactions	packetbeat-*

cc @rasroh

[LeeDr]

Some crappy code that generated the table above;

import {
  bdd,
  esClient
} from '../../support';

import PageObjects from '../../support/page_objects';

var expect = require('expect.js');

bdd.describe('dashboard sources', function () {
  const watchId = 'cluster_health_watch_' + new Date().getTime();
  const kbnInternVars = global.__kibana__intern__;
  const config = kbnInternVars.intern.config;

  bdd.it('should show the each dashboard with its visualizations, searches, and indexes', function () {
    var dashMap = {};
    var visMap = {};
    var searchMap = {};
    var indexMap = {};
    var w1 = 40;
    var w2 = 55;
    var w3 = 50;
    var w4 = 20;

    return esClient.search('.kibana', 'index-pattern')
    .then((response) => {

      function getIndexSource(index) {
        indexMap[index._id] = index._source;
      }

      var getChartTypesPromises4 = response.hits.hits.map(getIndexSource);
      return Promise.all(getChartTypesPromises4)
      .then((response) => {
        return esClient.search('.kibana', 'visualization')
        .then((response) => {

          function getVisualizationSource(vis) {
            visMap[vis._id] = vis._source;
          }

          var getChartTypesPromises2 = response.hits.hits.map(getVisualizationSource);
          return Promise.all(getChartTypesPromises2);
        });
      })
      .then((response) => {
        return esClient.search('.kibana', 'search')
        .then((response) => {

          function getSearchSource(search) {
            searchMap[search._id] = search._source;
          }
          var getChartTypesPromises3 = response.hits.hits.map(getSearchSource);
          return Promise.all(getChartTypesPromises3);
        })
        .then((response) => {
          return esClient.search('.kibana', 'dashboard')
          .then((response) => {

            function getDashboardSource(dashboard) {
              dashMap[dashboard._id] = dashboard._source;
              // console.log('| ' + formatString('*Dashboard*',w1) + ' | '
              //   + formatString('*Visualization*', w2) + ' | '
              //   + formatString('*Search*', w3) + ' | '
              //   + formatString('*Index Pattern*', w4) + ' | ');


              function printVis(vis) {
                if (vis.type === 'visualization') {
                  if (visMap[vis.id].savedSearchId) {
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString(searchMap[visMap[vis.id].savedSearchId].title, w3) + ' | '
                      + formatString(JSON.parse(searchMap[visMap[vis.id].savedSearchId].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                  } else if (JSON.parse(visMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index) {
                    // it's built directly on an index, or is Markdown
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString('      ', w3) + ' | '
                      + formatString(JSON.parse(visMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                  } else {
                    // markdown ?
                    console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                      + formatString(visMap[vis.id].title, w2) + ' | '
                      + formatString('      ', w3) + ' | '
                      + formatString('      ', w4) + ' | ');
                  }
                } else if (vis.type === 'search') {
                  console.log('| ' + formatString(dashboard._source.title,w1) + ' | '
                  + formatString('      ', w2) + ' | '
                    + formatString(searchMap[vis.id].title, w3) + ' | '
                    + formatString(JSON.parse(searchMap[vis.id].kibanaSavedObjectMeta.searchSourceJSON).index, w4) + ' | ');
                }
              }

              var vis = JSON.parse(dashboard._source.panelsJSON).map(printVis);
              return Promise.all(vis);

            }

            var getChartTypesPromises = response.hits.hits.map(getDashboardSource);
            return Promise.all(getChartTypesPromises);
          });
        });

      });
    });

  });

  function formatString(str, width) {
    return (str + '                                                    ').substring(0, width);
  }

});

[LeeDr]

We now have a relationship API that should make this easier?

[elasticmachine]

Pinging @elastic/kibana-platform