[Monitoring] Rename ccr fields based on changes in ES #24519

chrisronline · 2018-10-24T17:02:57Z

Relates to elastic/elasticsearch#34703 and #23013

NOTE: This PR will fail CI and not work until the appropriate ES PR is merged. elastic/elasticsearch#34836

This PR updates the UI code to account for field names changing in ES. The changed fields are located here: elastic/elasticsearch#34703 (comment)

elasticmachine · 2018-10-24T19:15:03Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

bleskes · 2018-10-25T07:19:14Z

Here's the relevant PR on the ES side: elastic/elasticsearch#34836

bleskes

Thanks @chrisronline . I left some comments. I also wanted to double check that the ui change to accomodate the move from a leader cluster to a remote cluster?

bleskes · 2018-10-25T07:20:26Z

x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/ccr_shard.js

@@ -32,7 +32,7 @@ async function getCcrStat(req, esIndexPattern, filters) {
    filterPath: [
      'hits.hits._source.ccr_stats',
      'hits.hits._source.timestamp',
-      'hits.hits.inner_hits.oldest.hits.hits._source.ccr_stats.number_of_operations_indexed',
+      'hits.hits.inner_hits.oldest.hits.hits._source.ccr_stats.operations_read',


this should be operations_written.

bleskes · 2018-10-25T07:20:50Z

x-pack/plugins/monitoring/server/routes/api/v1/elasticsearch/ccr_shard.js

@@ -32,7 +32,7 @@ async function getCcrStat(req, esIndexPattern, filters) {
    filterPath: [
      'hits.hits._source.ccr_stats',
      'hits.hits._source.timestamp',
-      'hits.hits.inner_hits.oldest.hits.hits._source.ccr_stats.number_of_operations_indexed',
+      'hits.hits.inner_hits.oldest.hits.hits._source.ccr_stats.operations_read',
      'hits.hits.inner_hits.oldest.hits.hits._source.ccr_stats.number_of_failed_fetches',


this one should be failed_read_requests

bleskes · 2018-10-25T07:21:17Z

x-pack/test/api_integration/apis/monitoring/elasticsearch/fixtures/ccr_shard.json

  },
  "timestamp": "2018-09-19T20:01:00.440Z",
  "oldestStat": {
    "number_of_failed_fetches": 0,
-    "number_of_operations_indexed": 1
+    "operations_read": 1


operations_written

bleskes · 2018-10-25T07:21:38Z

x-pack/test/api_integration/apis/monitoring/elasticsearch/fixtures/ccr_shard.json

-    "number_of_operations_indexed": 86,
-    "fetch_exceptions": [],
-    "time_since_last_fetch_millis": 19886
+    "operations_read": 86,


operations_written

chrisronline · 2018-10-25T14:11:09Z

Update here.

I've verified that all tests pass locally but they will most likely continue to fail on CI until the snapshot the CI tool uses for ES is updated to reflect the latest changes on the ES side (@ruflin kicked off the build so it should be soon-ish).

I propose we skip the tests that are failing in this PR so we can get green CI, then merge it in. I'll then open another PR where these tests are re-added and we merge that once CI is using the right snapshot and the tests are passing in the CI environment.

Here are the tests passing locally with the latest ES build:

Elastic-MBP:x-pack chris$ node ../scripts/functional_test_runner.js --config test/api_integration/config.js --grep "ccr"
 info Config loaded
 info Starting tests

 └-: apis
   └-> "before all" hook
   └-: Monitoring
     └-> "before all" hook
     └-: Elasticsearch
       └-> "before all" hook
       └-: ccr
         └-> "before all" hook
         └-> "before all" hook: load archive
           │ info [monitoring/ccr] Loading "mappings.json"
           │ info [monitoring/ccr] Loading "data.json.gz"
           │ info [monitoring/ccr] Created index ".monitoring-es-6-2018.09.19"
           │ info [monitoring/ccr] Indexed 3335 docs into ".monitoring-es-6-2018.09.19"
         └-> should return all followers and a grouping of stats by follower index
           └-> "before each" hook: global before each
           └- ✓ pass 
         └-> "after all" hook: unload archive
           │ info [monitoring/ccr] Unloading indices from "mappings.json"
           │ info [monitoring/ccr] Deleted existing index ".monitoring-es-6-2018.09.19"
           │ info [monitoring/ccr] Unloading indices from "data.json.gz"
         └-> "after all" hook
       └-: ccr shard
         └-> "before all" hook
         └-> "before all" hook: load archive
           │ info [monitoring/ccr] Loading "mappings.json"
           │ info [monitoring/ccr] Loading "data.json.gz"
           │ info [monitoring/ccr] Created index ".monitoring-es-6-2018.09.19"
           │ info [monitoring/ccr] Indexed 3335 docs into ".monitoring-es-6-2018.09.19"
         └-> should return specific shard details
           └-> "before each" hook: global before each
           └- ✓ pass 
         └-> "after all" hook: unload archive
           │ info [monitoring/ccr] Unloading indices from "mappings.json"
           │ info [monitoring/ccr] Deleted existing index ".monitoring-es-6-2018.09.19"
           │ info [monitoring/ccr] Unloading indices from "data.json.gz"
         └-> "after all" hook
       └-> "after all" hook
     └-: common
       └-> "before all" hook
       └-: mappings
         └-> "before all" hook
         └-> "before all" hook: load mappings
         └-> "before all" hook: load mappings
         └-> "before all" hook: load mappings
         └-> "before all" hook: load mappings
         └-> "before all" hook: load mappings
         └-: for es metrics
           └-> "before all" hook
           └-> ccr_stats.time_since_last_read_millis should exist in the mappings
             └-> "before each" hook: global before each
             └- ✓ pass 
           └-> ccr_stats.leader_max_seq_no should exist in the mappings
             └-> "before each" hook: global before each
             └- ✓ pass 
           └-> ccr_stats.follower_global_checkpoint should exist in the mappings
             └-> "before each" hook: global before each
             └- ✓ pass 
           └-> "after all" hook
         └-> "after all" hook
       └-> "after all" hook
     └-> "after all" hook
   └-> "after all" hook


5 passing (4.0s)

chrisronline · 2018-10-25T14:22:43Z

I talked with @bleskes @ruflin and @ycombinator about merging this PR before waiting for CI and we're in favor of it.

Pinging @epixa for thoughts

elasticmachine · 2018-10-25T16:30:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* Rename ccr fields based on changes in ES * More renames * Update archive data * Update snapshot * Skip the api integration tests for now

chrisronline · 2018-10-25T17:07:15Z

Backport:

6.x: 2182e27

chrisronline · 2018-10-25T17:51:43Z

PR to re-enable tests: #24600

Rename ccr fields based on changes in ES

0502b0c

chrisronline added review v7.0.0 Team:Monitoring Stack Monitoring team v6.5.0 labels Oct 24, 2018

chrisronline self-assigned this Oct 24, 2018

bleskes suggested changes Oct 25, 2018

View reviewed changes

chrisronline added 3 commits October 25, 2018 08:54

More renames

59cbebc

Update archive data

8cdcdb4

Update snapshot

93169cb

Skip the api integration tests for now

1b75311

bleskes approved these changes Oct 25, 2018

View reviewed changes

rashmivkulkarni added the blocker label Oct 25, 2018

chrisronline merged commit af3e621 into elastic:master Oct 25, 2018

chrisronline deleted the monitoring/ccr_rename_fields branch October 25, 2018 16:31

chrisronline mentioned this pull request Oct 25, 2018

[6.x] [Monitoring] Rename ccr fields based on changes in ES (#24519) #24593

Merged

chrisronline added a commit that referenced this pull request Oct 25, 2018

[Monitoring] Rename ccr fields based on changes in ES (#24519) (#24593)

2182e27

* Rename ccr fields based on changes in ES * More renames * Update archive data * Update snapshot * Skip the api integration tests for now

chrisronline mentioned this pull request Oct 25, 2018

[Monitoring] Re-enable CCR monitoring tests #24600

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] Rename ccr fields based on changes in ES #24519

[Monitoring] Rename ccr fields based on changes in ES #24519

chrisronline commented Oct 24, 2018 •

edited

elasticmachine commented Oct 24, 2018

bleskes commented Oct 25, 2018

bleskes left a comment •

edited

bleskes Oct 25, 2018

bleskes Oct 25, 2018

bleskes Oct 25, 2018

bleskes Oct 25, 2018

chrisronline commented Oct 25, 2018

chrisronline commented Oct 25, 2018

elasticmachine commented Oct 25, 2018

chrisronline commented Oct 25, 2018

chrisronline commented Oct 25, 2018

[Monitoring] Rename ccr fields based on changes in ES #24519

[Monitoring] Rename ccr fields based on changes in ES #24519

Conversation

chrisronline commented Oct 24, 2018 • edited

elasticmachine commented Oct 24, 2018

💔 Build Failed

bleskes commented Oct 25, 2018

bleskes left a comment • edited

Choose a reason for hiding this comment

bleskes Oct 25, 2018

Choose a reason for hiding this comment

bleskes Oct 25, 2018

Choose a reason for hiding this comment

bleskes Oct 25, 2018

Choose a reason for hiding this comment

bleskes Oct 25, 2018

Choose a reason for hiding this comment

chrisronline commented Oct 25, 2018

chrisronline commented Oct 25, 2018

elasticmachine commented Oct 25, 2018

💚 Build Succeeded

chrisronline commented Oct 25, 2018

chrisronline commented Oct 25, 2018

chrisronline commented Oct 24, 2018 •

edited

bleskes left a comment •

edited