[Monitoring] Kibana settings api #21100

chrisronline · 2018-07-23T17:56:20Z

Resolves #20903

Summary

This PR exposes an api that includes nearly the same information from the kibana_settings monitoring docs through an api endpoint for consumption through Metricbeat.

The api endpoint is: /api/settings and the response looks like:

{
  "cluster_uuid": "iB7C1tpwSamLc-ye8qTEwA",
  "settings": {
    "kibana": {
      "uuid": "5b2de169-2785-441b-ae8c-186a1936b17d",
      "name": "Elastic-MBP.local",
      "index": ".kibana",
      "host": "localhost",
      "transport_address": "localhost:5601",
      "version": "7.0.0-alpha1",
      "snapshot": false,
      "status": "green"
    }
  }
}

Testing

To test, you'll need to grab sample kibana_settings monitoring document:

POST .monitoring-kibana-6-*/_search
{
  "_source": "kibana_settings",
  "sort": {
    "timestamp": "desc"
  },
  "query": {
    "term": {
      "type": {
        "value": "kibana_settings"
      }
    }
  }
}

Compare the response from the above (within the kibana field) with the kibana field from the api response and ensure they match.

elasticmachine · 2018-07-23T20:23:18Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

ycombinator · 2018-07-26T21:34:56Z

Functionally this LGTM. I tested it with elastic/beats#7664 and it works as expected. Will review code shortly.

ycombinator · 2018-07-26T21:36:25Z

x-pack/plugins/xpack_main/server/routes/api/v1/settings.js

+        const settings = await settingsCollector.fetch(callCluster);
+        const uuid = await getClusterUuid(callCluster);
+
+        console.log('uuid', uuid);


Debugging statement.

ycombinator

Just a minor comment, and then LGTM.

elasticmachine · 2018-07-26T23:32:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

ycombinator · 2018-07-26T23:36:07Z

x-pack/plugins/xpack_main/server/routes/api/v1/settings.js

+ */
+
+import { wrap as wrapError } from 'boom';
+import { getKibanaInfoForStats } from '../../../../../../../src/server/status/lib';


Looks like CI is failing on this line:

15:25:10 │ debg error [22:25:10.553] [fatal] Error: Cannot find module '../../../../../../../src/server/status/lib' 15:25:10 │ debg at Function.Module._resolveFilename (module.js:547:15) 15:25:10 │ debg at Function.Module._load (module.js:474:25) 15:25:10 │ debg at Module.require (module.js:596:17) 15:25:10 │ debg at require (internal/module.js:11:18) 15:25:10 │ debg at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-x-pack/kibana/build/kibana/node_modules/x-pack/plugins/xpack_main/server/routes/api/v1/settings.js:8:1) ...

elasticmachine · 2018-07-27T01:56:18Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

ycombinator · 2018-07-27T13:22:40Z

x-pack/plugins/xpack_main/server/routes/api/v1/settings.js

+          }
+        });
+      } catch(err) {
+        req.log(['error'], err); // FIXME doesn't seem to log anything useful if ES times out


Was this FIXME for addressing in this PR itself or a follow up PR?

Good question. It's a comment copied over from the other api endpoint. Let's say it will be addressed in a follow up PR

I suspect the follow PR would need to address this problem everywhere

ycombinator

Tested with elastic/beats#7664. LGTM.

elasticmachine · 2018-07-27T22:10:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-07-30T17:06:55Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…settings_api

elasticmachine · 2018-07-30T19:20:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

chrisronline · 2018-07-31T14:35:12Z

@simianhacker Have some cycles to review this for me?

simianhacker

LGTM... I had a question about the tests. Right now you're just making sure at least one cluster show's up but that doesn't really test if we support more then one.

simianhacker · 2018-08-06T18:32:50Z

x-pack/plugins/xpack_main/server/routes/api/v1/settings.js

+          }
+        });
+      } catch(err) {
+        req.log(['error'], err); // FIXME doesn't seem to log anything useful if ES times out


I suspect the follow PR would need to address this problem everywhere

simianhacker · 2018-08-06T18:37:39Z

x-pack/test/api_integration/apis/xpack_main/settings/settings.js

+          .get('/api/settings')
+          .set('kbn-xsrf', 'xxx')
+          .expect(200);
+        expect(body.cluster_uuid.length > 0).to.eql(true);


Shouldn't this be more then 1 if we expect multiple?

Yup, good point!

…settings_api

simianhacker · 2018-08-07T15:59:24Z

FYI... You can probably merge this once it's green.

elasticmachine · 2018-08-07T17:07:52Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Kibana settings api * Use different version of this utility * Adding settings api test * Fix these tests * Update test

chrisronline · 2018-08-07T18:49:45Z

Backport:

6.x: cf03af7

Resolves #7621. Depends on elastic/kibana#21100. X-Pack Monitoring of Kibana requires two types of documents in the `.monitoring-kibana-*` indices: `kibana_stats` and `kibana_settings`. We made Metricbeat's `kibana/stats` metricset index `kibana_stats` documents into `.monitoring-kibana-*` in #7525. This PR makes the same metricset index `kibana_settings` documents into `.monitoring-kibana-*`.

Resolves elastic#7621. Depends on elastic/kibana#21100. X-Pack Monitoring of Kibana requires two types of documents in the `.monitoring-kibana-*` indices: `kibana_stats` and `kibana_settings`. We made Metricbeat's `kibana/stats` metricset index `kibana_stats` documents into `.monitoring-kibana-*` in elastic#7525. This PR makes the same metricset index `kibana_settings` documents into `.monitoring-kibana-*`. (cherry picked from commit 2af5ab9)

Resolves #7621. Depends on elastic/kibana#21100. X-Pack Monitoring of Kibana requires two types of documents in the `.monitoring-kibana-*` indices: `kibana_stats` and `kibana_settings`. We made Metricbeat's `kibana/stats` metricset index `kibana_stats` documents into `.monitoring-kibana-*` in #7525. This PR makes the same metricset index `kibana_settings` documents into `.monitoring-kibana-*`. (cherry picked from commit 2af5ab9)

tsullivan · 2018-08-16T20:51:21Z

x-pack/test/api_integration/apis/xpack_main/settings/settings.js

+        return esArchiver.unload(archive);
+      });
+
+      it('should load multiple clusters', async () => {


description should be different :)

Kibana settings api

83e66f4

chrisronline self-assigned this Jul 23, 2018

ycombinator mentioned this pull request Jul 23, 2018

Report kibana_settings to X-Pack Monitoring elastic/beats#7664

Merged

ycombinator reviewed Jul 26, 2018

View reviewed changes

ycombinator requested changes Jul 26, 2018

View reviewed changes

ycombinator reviewed Jul 26, 2018

View reviewed changes

Use different version of this utility

5a34929

ycombinator reviewed Jul 27, 2018

View reviewed changes

Adding settings api test

63d494d

chrisronline changed the title ~~[WIP] [Monitoring] Kibana settings api~~ [Monitoring] Kibana settings api Jul 27, 2018

chrisronline added review v7.0.0 Team:Monitoring Stack Monitoring team v6.5.0 labels Jul 27, 2018

chrisronline requested a review from tsullivan July 27, 2018 20:51

ycombinator approved these changes Jul 27, 2018

View reviewed changes

Fix these tests

f04fe04

Merge remote-tracking branch 'elastic/master' into monitoring/kibana_…

62a59b3

…settings_api

chrisronline requested review from simianhacker and removed request for tsullivan July 31, 2018 14:35

simianhacker approved these changes Aug 6, 2018

View reviewed changes

Merge remote-tracking branch 'elastic/master' into monitoring/kibana_…

023bb6a

…settings_api

Update test

da82a4b

chrisronline merged commit 871d283 into elastic:master Aug 7, 2018

chrisronline deleted the monitoring/kibana_settings_api branch August 7, 2018 17:12

chrisronline added a commit to chrisronline/kibana that referenced this pull request Aug 7, 2018

[Monitoring] Kibana settings api (elastic#21100)

4fd1787

* Kibana settings api * Use different version of this utility * Adding settings api test * Fix these tests * Update test

chrisronline mentioned this pull request Aug 7, 2018

[6.x] [Monitoring] Kibana settings api (#21100) #21742

Merged

chrisronline added a commit that referenced this pull request Aug 7, 2018

[Monitoring] Kibana settings api (#21100) (#21742)

cf03af7

* Kibana settings api * Use different version of this utility * Adding settings api test * Fix these tests * Update test

ycombinator mentioned this pull request Aug 10, 2018

Report kibana_settings to X-Pack Monitoring (#7664) elastic/beats#7943

Merged

tsullivan mentioned this pull request Aug 16, 2018

Allow kibana_settings collector to return nothing #22091

Merged

tsullivan reviewed Aug 16, 2018

View reviewed changes

tsullivan mentioned this pull request Aug 28, 2018

Provide a Kibana Stats API that can be consumed by a Kibana Metricbeat metricset giving 1:1 parity with current Kibana Monitoring data #20393

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] Kibana settings api #21100

[Monitoring] Kibana settings api #21100

chrisronline commented Jul 23, 2018 •

edited

elasticmachine commented Jul 23, 2018

ycombinator commented Jul 26, 2018 •

edited

ycombinator Jul 26, 2018

ycombinator left a comment

elasticmachine commented Jul 26, 2018

ycombinator Jul 26, 2018

elasticmachine commented Jul 27, 2018

ycombinator Jul 27, 2018

chrisronline Jul 27, 2018

simianhacker Aug 6, 2018

ycombinator left a comment

elasticmachine commented Jul 27, 2018

elasticmachine commented Jul 30, 2018

elasticmachine commented Jul 30, 2018

chrisronline commented Jul 31, 2018

simianhacker left a comment

simianhacker Aug 6, 2018

simianhacker Aug 6, 2018

chrisronline Aug 7, 2018

simianhacker commented Aug 7, 2018

elasticmachine commented Aug 7, 2018

chrisronline commented Aug 7, 2018

tsullivan Aug 16, 2018

[Monitoring] Kibana settings api #21100

[Monitoring] Kibana settings api #21100

Conversation

chrisronline commented Jul 23, 2018 • edited

Summary

Testing

elasticmachine commented Jul 23, 2018

💔 Build Failed

ycombinator commented Jul 26, 2018 • edited

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 26, 2018

💔 Build Failed

Choose a reason for hiding this comment

elasticmachine commented Jul 27, 2018

💚 Build Succeeded

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 27, 2018

💔 Build Failed

elasticmachine commented Jul 30, 2018

💔 Build Failed

elasticmachine commented Jul 30, 2018

💚 Build Succeeded

chrisronline commented Jul 31, 2018

simianhacker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simianhacker commented Aug 7, 2018

elasticmachine commented Aug 7, 2018

💚 Build Succeeded

chrisronline commented Aug 7, 2018

Choose a reason for hiding this comment

chrisronline commented Jul 23, 2018 •

edited

ycombinator commented Jul 26, 2018 •

edited