[Maps] Blended layer that switches between documents and clusters #57879

nreese · 2020-02-18T16:40:31Z

This PR adds a new layer type, BLENDED_VECTOR. The layer type will dynamically show clusters or raw documents depending on how many features are visible in the current view box.

You can create a blended layer by selected Documents source and then selecting the dynamic clustering option under the new Scaling section.

elasticmachine · 2020-02-18T16:40:33Z

Pinging @elastic/kibana-gis (Team:Geo)

nreese · 2020-02-19T16:31:11Z

@thomasneirynck @jsanz I added you for review even though this PR is still WIP. From a Proof of Concept perspective, the PR is complete. Could you please review the functionality from a usability perspective. Then, if this is the direction we think we should continue, I can finish up the PR and get it ready for code review.

alexfrancoeur · 2020-02-19T22:06:08Z

Hey @nreese, this functionality is awesome. I have some comments around the UX and would be interested in hearing thoughts from you and the group on them.

The current implementation generally feels a bit hidden for the functionality it offers
Explicitly calling out the 10k limit seemed odd, that's not something most users are aware of initially. If we do feel like we need to present this, it might be better to make it more subtle. Maybe as subtext, an info button or something in addition to the label it self
There might also be value in customizing this limit, maxing out at 10k
The other functionality that was missing for me was the option to configure my "clustering", "grid" or "heatmap". We default to clusters, but if we're providing this blended option, maybe we should have similar controls that we do in the grid agg layer.
To me, it almost feels like this functionality should be the default experience for adding a layer. Any new user would probably expect this experience and any mature maps user, would want to customize and have more control over when / how layers are shown
I wonder if it makes sense to approach this change with a broader change to layers. I know we have had discussions around grouping layer types or minimizing the selection available. Some high level Elasticsearch layer with a selection for "Blended", "Heatmap", "Cluster", "Shapes", "Points", where "Blended" is the default layer of choice could be interesting. It might be worth having a brainstorm session with design on this.

Should we sync live on this? I'm always one for progress over perfection, but if we want this layer to be used I think we need find more ways to expose the functionality.

nreese · 2020-02-19T22:19:01Z

I wonder if it makes sense to approach this change with a broader change to layers. I know we have had discussions around grouping layer types or minimizing the selection available. Some high level Elasticsearch layer with a selection for "Blended", "Heatmap", "Cluster", "Shapes", "Points", where "Blended" is the default layer of choice could be interesting. It might be worth having a brainstorm session with design on this

We had a sync on this and decided the best course of action given existing saved objects was to enhance the documents source. We did not want to redesign the entire source selection to get this feature in.

jsanz · 2020-02-20T15:10:14Z

I agree with @alexfrancoeur that it would be great to have the chance to select the type of aggregation we want for the low zoom levels. Also, the 10K limit feels much like an implementation detail and we could present to the user the configuration for the layer in a more visual way. Related with that, when the layer changes between modes we could have some visual indication after the layer name, as we do now for other situations.

Have you already put logic on the legend for the aggregated layer based on the document source definition? Something like:

No thematic symbology: count aggregation
Qualitative mapping by keyword field: top hits as per [Maps] add Top term aggregation #57875
Quantitative mapping: average aggregation

I see users wanting to do different things, so this should not be enforced but proposed, or maybe we can do nothing, and let the user decide how they want to symbolize the aggregation.

Also, if we decide to apply some defaults, and since we could potentially apply this based on document source properties I guess we would need to set up a priority for properties like: first color, then size, then label text. etc.

nreese · 2020-03-14T17:46:59Z

@elasticmachine merge upstream

thomasneirynck

I think this is really excellent addition. It's the first feature that explicitly proposes a solution to the scalability issues with the default ES response limits. It automates a work-flow we now expect users to perform manually, and it retains Kibana's real-time searchability and filtering.

as for earlier comments:

The value is pulled from index setting maxResultWindow so its not as arbitrary as it seems

Agreed that we can call out the explicit limit. But could we provide a link to the docs? I think it's fair to point users to this setting and nudge them to either increase/decrease this limit (e.g. in dev-console).

I would like to avoid making it the default capability for a few reasons.

I agree with @alexfrancoeur that a user would benefit from this being the default behavior. When the nr-of-docs exceeds 10k, the user already pays a usability-tax.

Either:

1. existing: the documents just get magically dropped from the view (which can create significant gaps in coverage).
1. proposed: the performance of the extra round-trip of the count request

When docs exceed 10k, I think it would benefit the add-source wizard to present this choice up-front. In practice, that entire "scaling"-card would be presented in the add-source. Of course, users can always course-correct and change this setting in the layer-editor.

Now is not the time to re-design the entire layer creation process.

+1. Let's follow up on this as separate effort (e.g. as a top-level ask, for example when adding templated layers for ML, APM, ...).

More feedback forthcoming, but just wanted to start participation on this.

nreese · 2020-03-16T15:23:49Z

When docs exceed 10k, I think it would benefit the add-source wizard to present this choice up-front. In practice, that entire "scaling"-card would be presented in the add-source. Of course, users can always course-correct and change this setting in the layer-editor.

agreed but I believe this should be done in a separate PR. Should I create an issue for this?

nreese · 2020-03-16T15:58:59Z

@elasticmachine merge upstream

thomasneirynck

initial comments

x-pack/legacy/plugins/maps/common/descriptor_types.d.ts

x-pack/legacy/plugins/maps/public/layers/sources/es_search_source/update_source_editor.js

thomasneirynck · 2020-03-16T21:03:33Z

x-pack/legacy/plugins/maps/public/layers/layer.js

@@ -333,10 +342,6 @@ export class AbstractLayer {
    return [];
  }

-  async getFields() {


x-pack/legacy/plugins/maps/public/layers/blended_vector_layer.ts

…type

thomasneirynck

lgtm. great feature!

it would be nice if we can work out details still in subsequent PRs:

improve wording of new "scaling"-card
move some (or all) of these scaling-settings into the add-source wizard so this functionality surfaces better (similar to how dynamic bounded-queries are added there)

x-pack/legacy/plugins/maps/public/layers/blended_vector_layer.ts

x-pack/legacy/plugins/maps/common/data_request_descriptor_types.d.ts

x-pack/legacy/plugins/maps/public/layers/layer.js

kibanamachine · 2020-03-18T17:58:56Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7d0dc6b

History

💚 Build #34059 succeeded fe3499f
💔 Build #33259 failed 56ef685
💔 Build #33254 failed 08b0fb4
💔 Build #33061 failed 4ab5e57
💔 Build #32940 failed b1650f4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…astic#57879) * [Maps] Blended layer that switches between documents and clusters * change layer type when scalingType changes * getSource * use cluster source when count exceeds value * ensure doc source stays in editor * start creating cluster style * pass all parts of style descriptor * get toggling between sources working * derive cluster style from document style * remove references to METRIC_TYPE * fix import * start typescripting blended_vector_layer * more typescript work * last of the TS errors * add migration to convert useTopTerm to scalingType * clean up * remove MapSavedObject work since its in a seperate PR now * fix EsSearchSource update editor jest test * fix map_selector jest test * move mutable state out of BlendedVectorLayer * one more change for removing mutable BlendedVectorLayer state * integrate newly merged MapSavedObjectAttributes type * review feedback * use data request for fetching feature count * add functional test * fix functional test * review feedback Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…7879) (#60550) * [Maps] Blended layer that switches between documents and clusters * change layer type when scalingType changes * getSource * use cluster source when count exceeds value * ensure doc source stays in editor * start creating cluster style * pass all parts of style descriptor * get toggling between sources working * derive cluster style from document style * remove references to METRIC_TYPE * fix import * start typescripting blended_vector_layer * more typescript work * last of the TS errors * add migration to convert useTopTerm to scalingType * clean up * remove MapSavedObject work since its in a seperate PR now * fix EsSearchSource update editor jest test * fix map_selector jest test * move mutable state out of BlendedVectorLayer * one more change for removing mutable BlendedVectorLayer state * integrate newly merged MapSavedObjectAttributes type * review feedback * use data request for fetching feature count * add functional test * fix functional test * review feedback Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

nreese added 8 commits February 13, 2020 15:06

[Maps] Blended layer that switches between documents and clusters

b369b1a

change layer type when scalingType changes

a1d170d

getSource

9eb460a

use cluster source when count exceeds value

3e5010d

ensure doc source stays in editor

ca06b90

start creating cluster style

433d693

pass all parts of style descriptor

6a3b194

get toggling between sources working

536e060

nreese added WIP Work in progress [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation enhancement New value added to drive a business result v8.0.0 v7.7.0 labels Feb 18, 2020

nreese requested a review from a team as a code owner February 18, 2020 16:40

nreese added release_note:enhancement and removed enhancement New value added to drive a business result labels Feb 18, 2020

This was referenced Feb 18, 2020

[Maps] blended ES document source that shows clusters and documents #48459

Closed

Clicking 'Visualize' on a location field in Discover should bring up new Maps app #57051

Closed

nreese added 2 commits February 18, 2020 14:32

merge with master

3c8e9bb

derive cluster style from document style

9a2b4d0

nreese requested review from jsanz and thomasneirynck February 19, 2020 16:28

thomasneirynck mentioned this pull request Feb 20, 2020

[Maps] add Top term aggregation #57875

Merged

nreese mentioned this pull request Feb 25, 2020

[Maps] direct Discover "visualize" to open Maps application #58549

Merged

Merge branch 'master' of github.com:elastic/kibana into scaling_type

554631c

fix map_selector jest test

56ef685

Merge branch 'master' into scaling_type

b62cdd3

thomasneirynck reviewed Mar 16, 2020

View reviewed changes

thomasneirynck self-requested a review March 16, 2020 03:20

Merge branch 'master' into scaling_type

b55c67d

thomasneirynck reviewed Mar 16, 2020

View reviewed changes

nreese added 6 commits March 17, 2020 09:21

Merge branch 'master' of github.com:elastic/kibana into scaling_type

f70c0ba

Merge branch 'scaling_type' of github.com:nreese/kibana into scaling_…

9dcb5eb

…type

move mutable state out of BlendedVectorLayer

11615d0

one more change for removing mutable BlendedVectorLayer state

7c5fa30

integrate newly merged MapSavedObjectAttributes type

6e70f51

review feedback

deca3c3

nreese requested a review from thomasneirynck March 17, 2020 17:17

nreese added 3 commits March 17, 2020 12:40

use data request for fetching feature count

1207873

add functional test

41a4bff

fix functional test

fe3499f

thomasneirynck approved these changes Mar 18, 2020

View reviewed changes

x-pack/legacy/plugins/maps/public/layers/blended_vector_layer.ts Show resolved Hide resolved

x-pack/legacy/plugins/maps/common/data_request_descriptor_types.d.ts Show resolved Hide resolved

x-pack/legacy/plugins/maps/public/layers/layer.js Outdated Show resolved Hide resolved

nreese added 2 commits March 18, 2020 10:12

review feedback

5f3236c

Merge branch 'master' of github.com:elastic/kibana into scaling_type

7d0dc6b

nreese merged commit 7e085ea into elastic:master Mar 18, 2020

nreese mentioned this pull request Mar 18, 2020

[7.x] [Maps] Blended layer that switches between documents and clusters (#57879) #60550

Merged

thomasneirynck moved this from In progress to Done in Maps Mar 18, 2020

nreese mentioned this pull request Mar 23, 2020

[Maps] fix point to point source regression #60930

Merged

nreese mentioned this pull request Apr 6, 2020

[Maps] automatically transition between aggregated view and individual documents in a single layer #46711

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] Blended layer that switches between documents and clusters #57879

[Maps] Blended layer that switches between documents and clusters #57879

nreese commented Feb 18, 2020 •

edited

Loading

elasticmachine commented Feb 18, 2020

nreese commented Feb 19, 2020

alexfrancoeur commented Feb 19, 2020

nreese commented Feb 19, 2020

jsanz commented Feb 20, 2020

nreese commented Mar 14, 2020

thomasneirynck left a comment •

edited

Loading

nreese commented Mar 16, 2020

nreese commented Mar 16, 2020

thomasneirynck left a comment

thomasneirynck Mar 16, 2020

thomasneirynck left a comment

kibanamachine commented Mar 18, 2020

[Maps] Blended layer that switches between documents and clusters #57879

[Maps] Blended layer that switches between documents and clusters #57879

Conversation

nreese commented Feb 18, 2020 • edited Loading

elasticmachine commented Feb 18, 2020

nreese commented Feb 19, 2020

alexfrancoeur commented Feb 19, 2020

nreese commented Feb 19, 2020

jsanz commented Feb 20, 2020

nreese commented Mar 14, 2020

thomasneirynck left a comment • edited Loading

Choose a reason for hiding this comment

nreese commented Mar 16, 2020

nreese commented Mar 16, 2020

thomasneirynck left a comment

Choose a reason for hiding this comment

thomasneirynck Mar 16, 2020

Choose a reason for hiding this comment

thomasneirynck left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 18, 2020

💚 Build Succeeded

History

nreese commented Feb 18, 2020 •

edited

Loading

thomasneirynck left a comment •

edited

Loading