[Remote clusters] Add support for proxy mode #59221

alisonelizabeth · 2020-03-03T21:14:39Z

Summary

Items to note:

There is a deprecated way to specify "proxy" mode for a remote cluster. This will not be handled by ES on upgrades, as it would involve deleting a user's settings, which is not ideal. The deprecated way was also undocumented. The UI attempts to detect if the cluster settings contain the deprecated setting, and warn the user. In edit mode, if the user saves the cluster, it will be saved using the new format. I have tested this with mocked data, but need to also coordinate with the cloud team to verify behavior.
Both the server_name and deprecated proxy field are not available via the GET /_remote/info API, so we need to access them via cluster settings. I only check persistent settings, based on a previous conversation here: [CCR] Remote clusters settings sources #26067 (comment). I tried to document this in the code, but please let me know if there are any questions or if it is unclear.
sniff (default) and proxy modes have different settings. To simplify the table view, I generalized the column names, e.g., "Addresses" maps to proxy_address for "proxy" mode and seeds for "sniff" mode. On the details panel, however, I use the more specific description, e.g., "Proxy address" and "Seeds".

Release note

The Remote Clusters UI added support for enabling "proxy" mode when creating or editing a remote cluster. This feature aligns with Elasticsearch's support for remote cluster connection modes that was added in 7.6.

How to test

You will need to set up two clusters in order to test remote clusters. For example:

yarn es snapshot
yarn es snapshot -E cluster.name="my_cluster" -E http.port=9201 -E path.data=/tmp/es (in another tab)

Test cases:

Create a remote cluster using the default mode
Create a remote cluster with the "proxy" mode enabled
Verify detail panel for both clusters
Verify edit and delete actions

Modify the responses in get_route.ts to simulate a cluster with deprecated settings, and verify icon displays in table and callout appears on the detail panel and edit route.

Example mock data:

    // [TESTING ONLY] mock remote cluster with deprecated settings
    clusterSettings.persistent.cluster.remote.deprecation_test = {
      proxy: '127.0.0.1:9300',
      seeds: ['127.0.0.1:9300'],
    };

  // [TESTING ONLY] mock remote cluster with deprecated settings
   clustersByName.deprecation_test = {
     seeds: ['127.0.0.1:9300'],
     connected: false,
     num_nodes_connected: 0,
     max_connections_per_cluster: 3,
     initial_connect_timeout: '30s',
     skip_unavailable: false,
   };

Screenshots

Proxy mode enabled

Deprecation warnings

elasticmachine · 2020-03-03T21:14:42Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

…e_clusters_proxy_mode

alisonelizabeth · 2020-03-13T19:22:32Z

@gchaps do you think you could review the copy in the screenshots I included in the PR description? I also will need to work with you to update the remote clusters documentation. At a minimum, the screenshot should be updated.

cjcenizal

This looks fantastic, Alison! I tested locally and everything works as expected. Code looks impeccable. Great job!

cjcenizal · 2020-03-13T22:51:40Z

x-pack/plugins/remote_clusters/common/lib/cluster_serialization.ts

-export function deserializeCluster(name: string, esClusterObject: any): any {
+import { PROXY_MODE } from '../constants';
+
+export interface ClusterEs {


Thanks for adding these types! 😬

cjcenizal · 2020-03-13T22:56:57Z

...e_clusters/public/application/sections/components/remote_cluster_form/remote_cluster_form.js

    } = this.state;

+    let modeSettings;


Great decision here, to allow for caching of the user's input if they flip between modes.

cjcenizal · 2020-03-13T23:01:04Z

...e_clusters/public/application/sections/components/remote_cluster_form/remote_cluster_form.js

-          <FormattedMessage
-            id="xpack.remoteClusters.remoteClusterForm.sectionSeedsDescription1"
-            defaultMessage="A list of remote cluster nodes to query for the cluster state.
-              Specify multiple seed nodes so discovery doesn't fail if a node is unavailable."


I feel like we lose a bit of helpful information here. Can we move some of this content into the help text below the "Seed nodes" input, changing it to: "An IP address or host name, followed by the transport port of the remote cluster. Specify multiple seed nodes so discovery doesn't fail if a node is unavailable."

Good point!

cjcenizal · 2020-03-13T23:02:25Z

...e_clusters/public/application/sections/components/remote_cluster_form/remote_cluster_form.js

+            <h2>
+              <FormattedMessage
+                id="xpack.remoteClusters.remoteClusterForm.sectionModeTitle"
+                defaultMessage="Connection mode settings"


I'll defer to @gchaps if she disagrees, but I think this can be simplified to just "Connection mode", since technically everything in this form is a setting.

cjcenizal · 2020-03-13T23:07:49Z

x-pack/plugins/remote_clusters/server/routes/api/get_route.ts

        // If the cluster hasn't been stored in the cluster state, then it's defined by the
        // node's config file.
        const isConfiguredByNode = !isTransient && !isPersistent;

+        // Pre-7.6, ES supported an undocumented "proxy" field
+        // ES does not handle migrating this to the new implementation, so we need to surface it in the UI
+        // This value is not available via the GET /_remote/info API, so we get it from the cluster settings


Thanks for these helpful comments.

cjcenizal · 2020-03-13T23:11:33Z

x-pack/plugins/remote_clusters/server/routes/api/update_route.ts

  skipUnavailable: schema.boolean(),
+  mode: schema.oneOf([schema.literal(PROXY_MODE), schema.literal(SNIFF_MODE)]),
+  seeds: schema.nullable(schema.arrayOf(schema.string())),
+  nodeConnections: schema.nullable(schema.number()),


Did we miss this in the original NP migration? I'm going to make a mental note to pay extra attention to this when I'm doing my own migration work because it seems super-easy to overlook a missed validation during testing. It seems the only way for us to discover a missed validation is with thorough functional or manual testing, right?

I actually added support for defining node connections as part of this PR. It was always possible via the ES API, but we were previously only allowing users to define seeds in the UI (screenshot).

Thanks, I missed that!

…e_clusters_proxy_mode

kibanamachine · 2020-03-16T19:28:48Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ca41b5f

History

💔 Build #33477 failed 91cd33c
💚 Build #33188 succeeded 4fe853f
💔 Build #33173 failed 1bb3b62
💔 Build #33153 failed 0c582ab
💔 Build #33088 failed eba69a6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

alisonelizabeth · 2020-03-16T19:46:10Z

@gchaps I'm going to go ahead and merge this PR, but I can address any copy feedback in a follow-up PR.

init commit: support for proxy mode

4fd3714

alisonelizabeth added release_note:enhancement Feature:CCR and Remote Clusters v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.7.0 labels Mar 3, 2020

alisonelizabeth added 17 commits March 4, 2020 15:44

fix edit remote clusters

a876d52

add proxy support in details panel

884731f

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

bd28e2f

…e_clusters_proxy_mode

update table

df590e3

fix validation

41a4397

cleanup: update tests, remove unused translations, add proxy badge

d84337d

fix ccr test

56671ca

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

e7aead7

…e_clusters_proxy_mode

update api tests

ecf78ff

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

3fa3cb4

…e_clusters_proxy_mode

add support for deprecated setting; add support for server_name

cbde9d1

add documentation link

de6393b

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

dcd5044

…e_clusters_proxy_mode

update tests

5a260c6

add callout to details panel for deprecated proxy setting

760073e

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

eba69a6

…e_clusters_proxy_mode

cleanup and test fixes

0c582ab

alisonelizabeth changed the title ~~[WIP] [Remote clusters] Add support for proxy mode~~ [Remote clusters] Add support for proxy mode Mar 13, 2020

alisonelizabeth marked this pull request as ready for review March 13, 2020 19:20

alisonelizabeth requested review from cjcenizal and gchaps March 13, 2020 19:20

cleanup detail panel logic

1bb3b62

update jest snapshot

4fe853f

cjcenizal approved these changes Mar 13, 2020

View reviewed changes

alisonelizabeth added 3 commits March 16, 2020 11:41

Merge branch 'master' of github.com:elastic/kibana into feature/remot…

95b2c96

…e_clusters_proxy_mode

address review feedback

91cd33c

update jest snapshots

ca41b5f

alisonelizabeth merged commit 77a859d into elastic:master Mar 16, 2020

alisonelizabeth deleted the feature/remote_clusters_proxy_mode branch March 16, 2020 19:46

alisonelizabeth mentioned this pull request Mar 16, 2020

[7.x] [Remote clusters] Add support for proxy mode (#59221) #60304

Merged

alisonelizabeth added a commit that referenced this pull request Mar 17, 2020

[Remote clusters] Add support for proxy mode (#59221) (#60304)

ad4ed17

alisonelizabeth mentioned this pull request Mar 17, 2020

[Remote clusters] Update copy #60382

Merged

alisonelizabeth mentioned this pull request Mar 24, 2020

[Remote clusters] Fix handling of remote clusters with deprecated proxy setting #61126

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote clusters] Add support for proxy mode #59221

[Remote clusters] Add support for proxy mode #59221

alisonelizabeth commented Mar 3, 2020 •

edited

elasticmachine commented Mar 3, 2020

alisonelizabeth commented Mar 13, 2020

cjcenizal left a comment

cjcenizal Mar 13, 2020

cjcenizal Mar 13, 2020

cjcenizal Mar 13, 2020

alisonelizabeth Mar 16, 2020

cjcenizal Mar 13, 2020

cjcenizal Mar 13, 2020

cjcenizal Mar 13, 2020

alisonelizabeth Mar 16, 2020 •

edited

cjcenizal Mar 16, 2020

kibanamachine commented Mar 16, 2020

alisonelizabeth commented Mar 16, 2020

[Remote clusters] Add support for proxy mode #59221

[Remote clusters] Add support for proxy mode #59221

Conversation

alisonelizabeth commented Mar 3, 2020 • edited

Summary

Release note

How to test

Screenshots

elasticmachine commented Mar 3, 2020

alisonelizabeth commented Mar 13, 2020

cjcenizal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alisonelizabeth Mar 16, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Mar 16, 2020

💚 Build Succeeded

History

alisonelizabeth commented Mar 16, 2020

alisonelizabeth commented Mar 3, 2020 •

edited

alisonelizabeth Mar 16, 2020 •

edited