World Countries and Administrative Regions update

[jsanz]

Opening the final PR to take into production the changes in the datasets developed on PRs #188 #192 and #195 that improve our World Countries and Admin Regions dataset.

Some remarks:

• The Administrative Regions dataset now includes country name and ISO code
• The new World Countries dataset:
  • is much more detailed but only available from 7.10.
  • contains population and area metric fields for future consumption by Kibana to help in data normalization processes.
  • is derived from the Administrative Regions dataset, merging regions from the same country
• The manifest for 7.10 includes both TopoJSON and GeoJSON versions of both datasets. TopoJSON is aimed for visualization purposes (Kibana) while GeoJSON is aimed for more generic consumption or analysis, like ingestion into Elasticsearch.

Check below a quick clip comparing the datasets between 7.8 and 7.10-SNAPSHOT.

These changes can be tested from the staging version in Kibana configuration or from EMS landing page.

map.emsFileApiUrl: https://storage.googleapis.com/elastic-bekitzur-emsfiles-vector-dev

7.8

7.10

cc. @thomasneirynck @nickpeihl

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[nickpeihl]

lgtm! tested in Kibana 7.10 and 7.9 with and without proxying in Kibana.

We probably discussed it, but I can't remember. What was our reasoning for exposing in the manifest the country codes in the administrative regions data? Tooltips? It doesn't look like we can filter on country codes on EMS Boundaries in Maps and I don't know why anyone would want to use them in joins.

[jsanz]

We probably discussed it, but I can't remember. What was our reasoning for exposing in the manifest the country codes in the administrative regions data? Tooltips? It doesn't look like we can filter on country codes on EMS Boundaries in Maps and I don't know why anyone would want to use them in joins.

I don't think we discussed it but my reasoning is that could be useful for anyone downloading it and ingesting it in the cluster. I think they can also be a good addition to any map tooltip showing regions

Finally, I also liked the idea of an explicit field linking both datasets instead of relying on the region iso2 prefix, but this is more speculative.

Still, if that's not something we'd like to offer we could simply remove the columns in the data processing after generating the countries dataset.

[thomasneirynck]

thx for all the updates. these are becoming really valuable, high quality datasets (and not just for Kibana users).

as discussed with @jsanz, I'd consider removing the metric-fields from the actual data (topo and geojson files) in the world-countries. Until there is support for these in Kibana ( in the UX (elastic/kibana#28367) and with supporting math-scripting (elastic/kibana#28265)) this is "phantom"-data that an end-user will not know is there.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[jsanz]

At ad846ee I've left the previous make targets and duplicated them without the metric processing so now the default target does not query World Bank or wikidata APIs to attach any area or population information. This will be easily reverted when we have metric support at Kibana and we are ready to advertise these type of fields.

[nickpeihl]

Still, if that's not something we'd like to offer we could simply remove the columns in the data processing after generating the countries dataset.

I think we should leave the columns in the data. I was more concerned about listing those fields in the manifest since they aren't very useful as join fields. But they are probably useful for tooltips so I think this is fine.