Region map visualization #10937

thomasneirynck · 2017-03-29T20:33:21Z

Summary: Kibana now has the Region Map Visualization. These are thematic maps in which boundary vector shapes are colored using a gradient, with higher intensity colors indicating larger values and lower intensity colors indicating smaller values. These are also known as choropleth maps. In order to color these layers, users specify a terms aggregation that matches a field in the vector layer. Kibana offers two vector layers by default; one for countries of the world and one for US Shapes. Users can also bring in their own vector layers by configuring the Kibana-configuration file to point to any GeoJson file that is hosted on a CORS-enabled server.

[EDIT]: see last comment for update wrt. data loading: #10937 (comment)

Introduce vector Map: Vector Maps / Choropleth Maps / Shapefile based fill maps #5039
rename map types: Rename map visualization types #11592

Feedback (round 2):

docs
play button shouldn't be enabled when configuration isn't valid
tests
get new icon (no longer use font-awesome)
make error messages consistent with Coordinate Map
desaturatization seems to be broken in coordinate map
manifest loading tests are broken due to refactor
ensure data isn't loaded twice

Feedback:

Todos:

add tests

thomasneirynck · 2017-04-10T06:33:48Z

src/ui/settings/defaults.js

+    'visualization:vectormap:showWarnings': {
+      value: true,
+      description: 'Should the vector map show a warning when terms cannot be joined to a shape on the map.'
+    },


not 100% sure of this.

Since we're mashing up 2 datasets, the join might not succeed (partially or completely). The odds of having one or more results not being present on the map can be high, especially when the data is not very clean. That's why I introduced an advanced setting to turn these warnings on/off.

ppisljar · 2017-04-10T07:57:57Z

src/core_plugins/choropleth/public/choropleth_controller.js

+import _ from 'lodash';
+import AggConfigResult from 'ui/vis/agg_config_result';
+import KibanaMap from 'ui/vis_maps/kibana_map';
+import FilterBarFilterBarClickHandlerProvider from 'ui/filter_bar/filter_bar_click_handler';


i think you might not actually need to import this, it should be available on vis.listeners.click

ppisljar · 2017-04-10T07:59:55Z

src/core_plugins/choropleth/public/choropleth_controller.js

+
+
+
+function getValue(metricsAgg, bucket) {


move on top ?

moved it up

ppisljar · 2017-04-10T08:00:20Z

src/core_plugins/choropleth/public/choropleth_layer.js

+    });
+
+    this._loaded = false;
+    $.ajax({//todo: replace with es6 fetch


ppisljar · 2017-04-10T08:25:38Z

this looks great! some small comments below:

default zoom level (0) shows world 5 times .... 2 seems more appropriate
metrics have no title, looks a bit empty
only one metric is possible, it should be expanded
add alpha3 and numeric iso country codes ?
documentation on how to add my custom vector data ?
shape field + aggregation type (term) should be pre selected as its the only possibility
2 letter country code instead of 2 letter abbreviation ?
parent pipeline aggregations should be excluded from the metric agg list
does geocentroid metric make sense ?

thomasneirynck · 2017-04-10T18:53:50Z

thanks @ppisljar , I'll get started on your feedback..

alexfrancoeur · 2017-04-11T13:59:11Z

@thomasneirynck finally had some time this morning to play around with vector map. Please see my comments below. I believe I mentioned some of these in the previous PR so they might already be on your radar.
Edit: I also now realize that I provided some duplicate feedback that Peter had. Sorry!

Map is still zoomed out when creating a vector map

When switching between state and country abbreviations, I lose tooltips until I refresh.
Is geo centroid meant to be an aggregation available for a vector map? And for that matter, all visualizations? I see it everywhere but tilemap at the moment.

Somewhat related, there are pipeline aggregations that need a histogram or date histogram in order to work. Can we remove these from vector map configurations if they cannot be used?

Also if you change the aggregation, the error does not go away.

I believe you already have this as a task but the custom labels are not working at the moment.

Having some issues generating reports locally but will try vector maps there as well.

kobelb

Nice job! Just a few comments, and one thing that can hopefully be removed.

I'm also unsure how I feel about calling it "choropleth" in code, but "vector map" to the end-user. I could see someone new to the project struggling to find the relevant code because of the mis-match in naming.

kobelb · 2017-04-12T13:10:36Z

test/functional/apps/visualize/index.js

-  require('./_heatmap_chart');
-  require('./_point_series_options');
-  require('./_shared_item');
+  // require('./_chart_types');


We'll want to revert this before committing.

kobelb · 2017-04-12T13:10:54Z

test/functional/index.js

-      'intern/dojo/node!./apps/xpack',
-      'intern/dojo/node!./apps/discover',
-      'intern/dojo/node!./apps/management',
+      // 'intern/dojo/node!./apps/xpack',


Same as above, we don't want to commit this to master

kobelb · 2017-04-12T13:20:14Z

test/functional/apps/visualize/_chart_types.js

@@ -25,6 +25,7 @@ bdd.describe('visualize app', function describeIndexTests() {
        'Data Table',
        'Metric',
        'Tile Map',
+        'Vector Map',


There are quite a few merge conflicts for the tests because of #10910

kobelb · 2017-04-12T14:07:13Z

src/core_plugins/choropleth/public/choropleth_controller.js

+  function getValue(metricsAgg, bucket) {
+    let size = metricsAgg.getValue(bucket);
+    if (typeof size !== 'number' || isNaN(size)) {
+      try {


I've been unable to find this code-path being hit, or other visualizations (besides tagcloud) doing something similar.

thomasneirynck · 2017-04-14T21:06:48Z

rebasing against master, incorporating test-suite changes

thomasneirynck · 2017-04-20T18:18:07Z

After more discussion, we want to change how we load the example json files. Instead of using Kibana to serve those, we should host them from a central repo. Apart from that change, users will continue to be able to bring in custom data layers with the same mechanism we have now.

thomasneirynck · 2017-04-26T22:13:57Z

removing default data config.

all custom data should now be configured in the .yml.

e.g.

vectormap:
  layers:
     - name: "World Countries"
       url: "../plugins/choropleth/data/world_countries.json"
       fields:
          - name: "name"
            description: "Country name"
          - name: "iso2"
            description: "Two letter abbreviation"
          - name: "iso2"
            description: "Three letter abbreviation"
     - name: "US States"
       url: "../plugins/choropleth/data/us_states.json"
       fields:
          - name: "postal"
            description: "Two letter abbreviation"
          - name: "name"
            description: "State name"

Default layers will now be loaded from remote catalogue service (forthcoming)

thomasneirynck · 2017-05-03T21:48:19Z

with 6b41e66, we're testing against a new metadata-service. This is just a POC now, the structure of these documents, and the endpoints are not final yet.

thomasneirynck · 2017-05-05T04:42:59Z

@ppisljar @kobelb fyi, I've refactored this to use a new file-server infra has setup (POC just now). These changes make all the unit tests fail, but do take an initial look already if you can. I'll work on tests once we've settled with infra on the final version of the new manifest schemas.

…ating vector-layers

kobelb

Functionally, this seems to work perfectly! Just a few comments.

Also, the optimize/.empty file looks like it was somehow deleted with this PR.

kobelb · 2017-05-31T11:23:07Z

q

@@ -0,0 +1,94 @@
+[1mdiff --git a/src/core_plugins/region_map/public/region_map_controller.js b/src/core_plugins/region_map/public/region_map_controller.js[m


Pretty sure this file was accidentally checked in.

kobelb · 2017-05-31T11:40:04Z

src/core_plugins/kibana/inject_vars.js

@@ -6,13 +8,20 @@ export function injectVars(server) {
  //keeping this logic for backward compatibilty.
  const configuredUrl = server.config().get('tilemap.url');
  const isOverridden = typeof configuredUrl === 'string' && configuredUrl !== '';
-  const tilemapConfig = serverConfig.get('tilemap');
+  const tilemapConfig = _.cloneDeep(serverConfig.get('tilemap'));


What's your reasoning for cloning these settings? It's my understanding that these injected_vars are serialized into an HTMLElement in the initial response and then deserialized client-side, so it seems that this might be unnecessary.

kobelb · 2017-05-31T11:40:14Z

src/core_plugins/kibana/inject_vars.js

@@ -6,13 +8,20 @@ export function injectVars(server) {
  //keeping this logic for backward compatibilty.
  const configuredUrl = server.config().get('tilemap.url');
  const isOverridden = typeof configuredUrl === 'string' && configuredUrl !== '';
-  const tilemapConfig = serverConfig.get('tilemap');
+  const tilemapConfig = _.cloneDeep(serverConfig.get('tilemap'));
+  const vectormapsConfig = _.cloneDeep(serverConfig.get('regionmap'));


A remnant of the vectormaps naming, should be regionmapsConfig.

kobelb · 2017-05-31T11:42:01Z

src/core_plugins/kibana/inject_vars.js

+  const mapConfig = _.cloneDeep(serverConfig.get('map'));
+
+
+  vectormapsConfig.layers =  (vectormapsConfig.layers) ? vectormapsConfig.layers : [];


Generally, I've seen us using the .default() method of Joi to set defaults for the config schema, so we could do something like the following in the 'src/server/config/schema.js' to do the equivalent:

regionmap: Joi.object({ layers: Joi.array().items(Joi.object({ url: Joi.string(), type: Joi.string(), name: Joi.string(), fields: Joi.array().items(Joi.object({ name: Joi.string(), description: Joi.string() })) })).default([]) }).default()

The .default([]) portion of above should always default the layers to an empty array if they aren't specified.

kobelb · 2017-05-31T11:55:32Z

src/core_plugins/region_map/public/choropleth_layer.js

+  }
+
+  const { min, max } = getMinMax(data);
+  data = data.slice();


Doing data.slice() here and then doing data.splice(lastIndex, 1) below inside of the getLeafletStyleFunction is rather confusing. We're using getLeafletStyleFunction to mutate data that is also used by getMismatches, so the order that these functions are called becomes significant. Was this intentional?

yes, that order is significant. the getMisMatches is just for the client to be able to read out the errors of the inner-join (ie. the style function).

kobelb · 2017-05-31T12:16:04Z

src/core_plugins/region_map/public/region_map_vis.js

+  const TemplateVisType = Private(TemplateVisTypeProvider);
+  const Schemas = Private(VisSchemasProvider);
+
+  const vectorLayers = vectormapsConfig.layers.slice();


This should only be being called once for the "client-side app" when it's registering all the visualization types, what are we trying to prevent with the slice()? I know before that we were having layers duplicated when creating multiple maps in a row, was this meant to address this?

kobelb · 2017-05-31T12:17:11Z

src/core_plugins/region_map/public/region_map_vis_params.js

+      link: function ($scope) {
+
+
+        console.log('calling the linker function...');


console.log should be removed before merging.

kobelb · 2017-05-31T12:23:01Z

src/server/config/schema.js

@@ -161,6 +161,13 @@ module.exports = () => Joi.object({
  status: Joi.object({
    allowAnonymous: Joi.boolean().default(false)
  }).default(),
+  map: Joi.object({
+    manifestServiceUrl: Joi.when('$dev', {


So the new map.manifesetServiceUrl is used for both the tiles and the geo layers, does that mean we should be deprecating the tilemap config section?

kobelb · 2017-05-31T12:23:46Z

src/server/config/schema.js

+  map: Joi.object({
+    manifestServiceUrl: Joi.when('$dev', {
+      is: true,
+      then: Joi.string().default('https://geo.elastic.co/v1/manifest'),


Dev and Non-Dev appear to be the same, was this intentional?

yes, for now this needs to be the same. Infra has only setup a single production server. this will likely change in the weeks to come.

kobelb · 2017-05-31T12:53:40Z

test/support/page_objects/visualize_page.js

@@ -0,0 +1,835 @@
+


This file's been moved, I think you should just be able to delete it.

rebase snafu, thanks.

ppisljar · 2017-05-31T13:48:22Z

sibling pipeline aggs should be allowed (no reason why they should be disabled i think)

sample use case:
i want to show usage of windows operating system around the world

for the shape field i select term aggregation on geo.src (to get countries)
for metric i would choose sum bucket aggregation where for bucket i would choose filter aggregation with filter like machine.os.raw: win* and for metric i would choose unique count on clientIp

i would get representation of windows usage around the world.

thomasneirynck · 2017-05-31T14:11:58Z

@ppisljar thanks, I added them.

ppisljar

LGTM

thomasneirynck · 2017-05-31T16:17:12Z

wrt. tilemap-config. still required, since people can override the url still to plug in a custom url. perhaps we can deprecate this configuration in 6.0, but keeping it for now.

kobelb · 2017-05-31T17:06:20Z

src/server/config/schema.js

+      then: Joi.string().default('https://geo.elastic.co/v1/manifest'),
+      otherwise: Joi.string().default('https://geo.elastic.co/v1/manifest')
+    })
+  }).default(),
  tilemap: Joi.object({
    manifestServiceUrl: Joi.when('$dev', {


This needs to go.

yep, sorry, mistake on my part. gone!

kobelb

LGTM

elastic#10937 seems to have brought this file back, likely as part of a bad merge resolution

#10937 seems to have brought this file back, likely as part of a bad merge resolution

elastic#10937 seems to have brought this file back, likely as part of a bad merge resolution

thomasneirynck changed the title ~~add vector map~~ Vector Map Mar 29, 2017

tbragin added the Feature:Visualizations Generic visualization features (in case no more specific feature label is available) label Apr 4, 2017

thomasneirynck force-pushed the vector_map branch 2 times, most recently from f60d556 to f597c30 Compare April 10, 2017 04:55

thomasneirynck requested review from kobelb and ppisljar April 10, 2017 06:20

thomasneirynck added the review label Apr 10, 2017

thomasneirynck commented Apr 10, 2017

View reviewed changes

ppisljar reviewed Apr 10, 2017

View reviewed changes

tbragin added v5.5.0 v6.0.0 labels Apr 10, 2017

kobelb reviewed Apr 12, 2017

View reviewed changes

thomasneirynck force-pushed the vector_map branch from 2353864 to 8e3e227 Compare April 14, 2017 21:05

thomasneirynck force-pushed the vector_map branch from 67dd87f to fe1f906 Compare April 26, 2017 19:53

thomasneirynck force-pushed the vector_map branch from 49744eb to 79284e4 Compare May 2, 2017 17:24

tbragin mentioned this pull request May 4, 2017

Rename map visualization types #11592

Closed

thomasneirynck force-pushed the vector_map branch from 6b41e66 to b19925d Compare May 5, 2017 22:11

thomasneirynck added the roadmap label May 12, 2017

thomasneirynck force-pushed the vector_map branch from b19925d to 1bf88c1 Compare May 12, 2017 15:32

thomasneirynck requested a review from kobelb May 12, 2017 21:25

thomasneirynck force-pushed the vector_map branch from 24cbb2a to a45b6b7 Compare May 31, 2017 03:43

move loading of vector-layers to directive/disable play button on upd…

91e2ffb

…ating vector-layers

kobelb reviewed May 31, 2017

View reviewed changes

add sibling aggregations

c739884

ppisljar approved these changes May 31, 2017

View reviewed changes

thomasneirynck added 2 commits May 31, 2017 12:14

fix rename

5c75704

remove log statement

91c5ac2

thomasneirynck added 2 commits May 31, 2017 12:21

remove unnecessary files

f462397

minor feedback

a4c3a5c

kobelb reviewed May 31, 2017

View reviewed changes

thomasneirynck added 2 commits May 31, 2017 13:09

improve naming/remove obsolete configs

ab4fe96

remove unused setting

fa93788

kobelb approved these changes May 31, 2017

View reviewed changes

thomasneirynck added 2 commits May 31, 2017 15:31

should respect overridden urls

5122ce1

remove test cruft

59cf9df

thomasneirynck merged commit 2270a6c into elastic:master May 31, 2017

w33ble mentioned this pull request Jun 2, 2017

Remove intern configuration #12149

Closed

epixa changed the title ~~Vector Map~~ Region Map Jun 2, 2017

epixa changed the title ~~Region Map~~ Region Map visualization Jun 2, 2017

epixa changed the title ~~Region Map visualization~~ Region map visualization Jun 2, 2017

w33ble added a commit to w33ble/kibana that referenced this pull request Jun 6, 2017

remove intern setup

d5ca1e4

elastic#10937 seems to have brought this file back, likely as part of a bad merge resolution

w33ble mentioned this pull request Jun 6, 2017

Remove intern setup #12201

Merged

w33ble added a commit that referenced this pull request Jun 6, 2017

remove intern setup (#12201)

c400d60

#10937 seems to have brought this file back, likely as part of a bad merge resolution

w33ble added a commit that referenced this pull request Jun 6, 2017

remove intern setup (#12201)

2a5dab1

#10937 seems to have brought this file back, likely as part of a bad merge resolution

chrisronline pushed a commit to chrisronline/kibana that referenced this pull request Jun 14, 2017

remove intern setup (elastic#12201)

5c8c65f

elastic#10937 seems to have brought this file back, likely as part of a bad merge resolution

thomasneirynck removed v6.0.0 v6.0.0-alpha2 labels Jul 11, 2017

		@@ -0,0 +1,94 @@
		[1mdiff --git a/src/core_plugins/region_map/public/region_map_controller.js b/src/core_plugins/region_map/public/region_map_controller.js[m

		const mapConfig = _.cloneDeep(serverConfig.get('map'));


		vectormapsConfig.layers = (vectormapsConfig.layers) ? vectormapsConfig.layers : [];

		link: function ($scope) {


		console.log('calling the linker function...');

Region map visualization #10937

Region map visualization #10937

Conversation

thomasneirynck commented Mar 29, 2017 • edited by jimgoodwin

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppisljar commented Apr 10, 2017

thomasneirynck commented Apr 10, 2017

alexfrancoeur commented Apr 11, 2017 • edited

kobelb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasneirynck commented Apr 14, 2017

thomasneirynck commented Apr 20, 2017

thomasneirynck commented Apr 26, 2017

thomasneirynck commented May 3, 2017

thomasneirynck commented May 5, 2017

kobelb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppisljar commented May 31, 2017 • edited

thomasneirynck commented May 31, 2017

ppisljar left a comment

Choose a reason for hiding this comment

thomasneirynck commented May 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kobelb left a comment

Choose a reason for hiding this comment

thomasneirynck commented Mar 29, 2017 •

edited by jimgoodwin

alexfrancoeur commented Apr 11, 2017 •

edited

ppisljar commented May 31, 2017 •

edited