Integrate GeoHexGridAggregation with vector tiles API (#84553)

This commit adds a new optional parameter on the vector tiles API called `grid_agg` with two
possible values, geotile (default) and geohex. This will allow to build the aggs layer using different
grid aggregations, for example we can have a grid aggregation that is built using hexagons.

docs/changelog/84553.yaml

@@ -0,0 +1,5 @@
+pr: 84553
+summary: Add `geohex_grid` aggregation to vector tiles API
+area: Geo
+type: enhancement
+issues: []

docs/reference/search/search-vector-tile-api.asciidoc

@@ -91,17 +91,19 @@ Internally, {es} translates a vector tile search API request into a
 * A <<query-dsl-geo-bounding-box-query,`geo_bounding_box`>> query on the
 `<field>`. The query uses the `<zoom>/<x>/<y>` tile as a bounding box.
 
-* A <<search-aggregations-bucket-geotilegrid-aggregation,`geotile_grid`>>
-aggregation on the `<field>`. The aggregation uses the `<zoom>/<x>/<y>` tile as
-a bounding box.
+* A <<search-aggregations-bucket-geotilegrid-aggregation,`geotile_grid`>> or
+<<search-aggregations-bucket-geohexgrid-aggregation,`geohex_grid`>> aggregation
+on the `<field>`. The `grid_agg` parameter determines the aggregation type. The
+aggregation uses the `<zoom>/<x>/<y>` tile as a bounding box.
 
 * Optionally, a
 <<search-aggregations-metrics-geobounds-aggregation,`geo_bounds`>> aggregation
 on the `<field>`. The search only includes this aggregation if the
 `exact_bounds` parameter is `true`.
 
-For example, {es} may translate a vector tile search API request with an
-`exact_bounds` argument of `true` into the following search:
+For example, {es} may translate a vector tile search API request with a
+`grid_agg` argument of `geotile` and an `exact_bounds` argument of `true`
+into the following search:
 
 [source,console]
 ----
@@ -159,21 +161,21 @@ Protobufs (PBF)]. By default, the tile contains three layers:
 * A `hits` layer containing a feature for each `<field>` value matching the
 `geo_bounding_box` query.
 
-*  An `aggs` layer containing a feature for each cell of the `geotile_grid`. You
-can use these cells as tiles for lower zoom levels. The layer only contains
-features for cells with matching data.
+*  An `aggs` layer containing a feature for each cell of the `geotile_grid` or
+`geohex_grid`. The layer only contains features for cells with matching data.
 
 * A `meta` layer containing:
 ** A feature containing a bounding box. By default, this is the bounding box of
 the tile.
-** Value ranges for any sub-aggregations on the `geotile_grid`.
+** Value ranges for any sub-aggregations on the `geotile_grid` or `geohex_grid`.
 ** Metadata for the search.
 
 The API only returns features that can display at its zoom level. For example,
 if a polygon feature has no area at its zoom level, the API omits it.
 
 The API returns errors as UTF-8 encoded JSON.
 
+[role="child_attributes"]
 [[search-vector-tile-api-query-params]]
 ==== {api-query-parms-title}
 
@@ -200,37 +202,132 @@ larger than the vector tile.
 square with equal sides. Defaults to `4096`.
 // end::extent-param[]
 
+// tag::grid-agg[]
+`grid_agg`::
+(Optional, string) Aggregation used to create a grid for the `<field>`.
++
+.Valid values for `grid_agg`
+[%collapsible%open]
+====
+`geotile` (Default)::
+<<search-aggregations-bucket-geotilegrid-aggregation,`geotile_grid`>>
+aggregation.
+
+`geohex`::
+<<search-aggregations-bucket-geohexgrid-aggregation,`geohex_grid`>> aggregation.
+If you specify this value, the `<field>` must be a <<geo-point,`geo_point`>>
+field.
+====
+// end::grid-agg[]
+
 // tag::grid-precision[]
 `grid_precision`::
-(Optional, integer) Additional zoom levels available through the `aggs` layer.
-For example, if `<zoom>` is `7` and `grid_precision` is `8`, you can zoom in up to
-level 15. Accepts `0`-`8`. Defaults to `8`. If `0`, results don't include the
-`aggs` layer.
-+
-This value determines the grid size of the `geotile_grid` as follows:
+(Optional, integer) Precision level for cells in the `grid_agg`. Accepts
+`0`-`8`. Defaults to `8`. If `0`, results don't include the `aggs` layer.
 +
+.Grid precision for `geotile`
+[%collapsible%open]
+====
+For a `grid_agg` of `geotile`, you can use cells in the `aggs` layer as tiles
+for lower zoom levels. `grid_precision` represents the additional zoom levels
+available through these cells. The final precision is computed by as
+follows:
+
+`<zoom> + grid_precision`
+
+For example, if `<zoom>` is `7` and `grid_precision` is `8`, then the
+`geotile_grid` aggregation will use a precision of `15`. The maximum final
+precision is `29`.
+
+The `grid_precision` also determines the number of cells for the grid as
+follows:
+
 `(2^grid_precision) x (2^grid_precision)`
-+
+
 For example, a value of `8` divides the tile into a grid of 256 x 256 cells. The
 `aggs` layer only contains features for cells with matching data.
+====
++
+.Grid precision for `geohex`
+[%collapsible%open]
+====
+For a `grid_agg` of `geohex`, {es} uses `<zoom>` and `grid_precision` to
+calculate a final precision as follows:
+
+`<zoom> + grid_precision`
+
+This precision determines the https://h3geo.org/docs/core-library/restable[H3
+resolution of the hexagonal cells] produced by the `geohex` aggregation. The 
+following table maps the H3 resolution for each precision.
+
+For example, if `<zoom>` is `3` and `grid_precision` is `3`, the precision is
+`6`. At a precision of `6`, hexagonal cells have an H3 resolution of `2`. If
+`<zoom>` is `3` and `grid_precision` is `4`, the precision is `7`. At a
+precision of `7`, hexagonal cells have an H3 resolution of `3`.
+
+[cols="<,<,<,<,<",options="header",]
+|====
+|Precision | Unique tile bins| H3 resolution| Unique hex bins | Ratio
+|1	|4	                 |0	|122	            |30.5
+|2	|16	                 |0	|122	            |7.625
+|3	|64	                 |1	|842	            |13.15625
+|4	|256	               |1	|842	            |3.2890625
+|5	|1024	               |2	|5882	            |5.744140625
+|6	|4096	               |2	|5882	            |1.436035156
+|7	|16384	             |3	|41162	          |2.512329102
+|8	|65536	             |3	|41162	          |0.6280822754
+|9	|262144	             |4	|288122	          |1.099098206
+|10	|1048576	           |4	|288122	          |0.2747745514
+|11	|4194304	           |5	|2016842	        |0.4808526039
+|12	|16777216	           |6	|14117882	        |0.8414913416
+|13	|67108864	           |6	|14117882	        |0.2103728354
+|14	|268435456	         |7	|98825162	        |0.3681524172
+|15	|1073741824	         |8	|691776122	      |0.644266719
+|16	|4294967296	         |8	|691776122	      |0.1610666797
+|17	|17179869184	       |9	|4842432842	      |0.2818666889
+|18	|68719476736	       |10 |33897029882	    |0.4932667053
+|19	|274877906944	       |11 |237279209162	  |0.8632167343
+|20	|1099511627776	     |11 |237279209162	  |0.2158041836
+|21	|4398046511104	     |12 |1660954464122	  |0.3776573213
+|22	|17592186044416	     |13 |11626681248842	|0.6609003122
+|23	|70368744177664	     |13 |11626681248842	|0.165225078
+|24	|281474976710656	   |14 |81386768741882	|0.2891438866
+|25	|1125899906842620	   |15 |569707381193162	|0.5060018015
+|26	|4503599627370500	   |15 |569707381193162	|0.1265004504
+|27	|18014398509482000	 |15 |569707381193162	|0.03162511259
+|28	|72057594037927900	 |15 |569707381193162	|0.007906278149
+|29	|288230376151712000	 |15 |569707381193162	|0.001976569537
+|====
+
+Hexagonal cells don't align perfectly on a vector tile. Some cells may intersect
+more than one vector tile. To compute the H3 resolution for each precision, {es}
+compares the average density of hexagonal bins at each resolution with the
+average density of tile bins at each zoom level. {es} uses the H3 resolution
+that is closest to the corresponding `geotile` density.
+====
 // end::grid-precision[]
 
 // tag::grid-type[]
 `grid_type`::
 (Optional, string) Determines the geometry type for features in the `aggs`
-layer. In the `aggs` layer, each feature represents a `geotile_grid` cell.
-Accepts:
-
-`grid` (Default):::
-Each feature is a `Polygon` of the cell's bounding box.
+layer. In the `aggs` layer, each feature represents a cell in the grid.
++
+.Valid values for `grid_type`
+[%collapsible%open]
+====
+`grid` (Default)::
+Each feature is a `Polygon` of the cell's geometry. For a `grid_agg` of
+`geotile`, the feature is the cell's bounding box. For a `grid_agg` of
+`geohex`, the feature is the hexagonal cell's boundaries.
 
-`point`:::
+`point`::
 Each feature is a `Point` that's the centroid of the cell.
 
-`centroid`:::
+`centroid`::
 Each feature is a `Point` that's the centroid of the data within the cell. For
 complex geometries, the actual centroid may be outside the cell. In these cases,
 the feature is set to the closest point to the centroid inside the cell.
+====
 // end::grid-type[]
 
 // tag::size[]
@@ -255,7 +352,7 @@ If `false`, the response does not include the total number of hits matching the
 
 `aggs`::
 (Optional, <<search-aggregations,aggregation object>>)
-<<run-sub-aggs,Sub-aggregations>> for the `geotile_grid`. Supports the following
+<<run-sub-aggs,Sub-aggregations>> for the `grid_agg`. Supports the following
 aggregation types:
 +
 * <<search-aggregations-metrics-avg-aggregation,`avg`>>
@@ -293,6 +390,8 @@ You can specify fields in the array as a string or object.
 include::search.asciidoc[tag=fields-param-props]
 ====
 
+include::search-vector-tile-api.asciidoc[tag=grid-agg]
+
 include::search-vector-tile-api.asciidoc[tag=grid-precision]
 
 include::search-vector-tile-api.asciidoc[tag=grid-type]
@@ -397,7 +496,7 @@ Field value. Only returned for fields in the `fields` parameter.
 ====
 
 `aggs`::
-(object) Layer containing results for the `geotile_grid` aggregation and its
+(object) Layer containing results for the `grid_agg` aggregation and its
 sub-aggregations.
 +
 .Properties of `aggs`
@@ -408,8 +507,7 @@ include::search-vector-tile-api.asciidoc[tag=extent]
 include::search-vector-tile-api.asciidoc[tag=version]
 
 `features`::
-(array of objects) Array of features. Contains a feature for each cell of the
-`geotile_grid`.
+(array of objects) Array of features. Contains a feature for each cell of the grid.
 +
 .Properties of `features` objects
 [%collapsible%open]
@@ -582,6 +680,7 @@ the `13/4207/2692` vector tile.
 ----
 GET museums/_mvt/location/13/4207/2692
 {
+  "grid_agg": "geotile",
   "grid_precision": 2,
   "fields": [
     "name",

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/vector-tile/10_basic.yml

@@ -88,6 +88,30 @@ setup:
         body:
           grid_type: point
 
+---
+"grid agg geotile":
+  - do:
+      search_mvt:
+        index: locations
+        field: location
+        x: 0
+        y: 0
+        zoom: 0
+        body:
+          grid_agg: geotile
+
+---
+"grid agg geohex":
+  - do:
+      search_mvt:
+        index: locations
+        field: location
+        x: 0
+        y: 0
+        zoom: 0
+        body:
+          grid_agg: geohex
+
 ---
 "grid type grid":
   - do:

x-pack/plugin/vector-tile/src/javaRestTest/java/org/elasticsearch/xpack/vectortile/VectorTileRestIT.java

@@ -317,10 +317,18 @@ public void testGridPrecision() throws Exception {
         }
     }
 
-    public void testGridType() throws Exception {
+    public void testGeoTileGrid() throws Exception {
+        doGridAggType(randomBoolean() ? "" : ", \"grid_agg\": \"geotile\"");
+    }
+
+    public void testGeoHexGrid() throws Exception {
+        doGridAggType(", \"grid_agg\": \"geohex\"");
+    }
+
+    private void doGridAggType(String gridAgg) throws Exception {
         {
             final Request mvtRequest = new Request(getHttpMethod(), INDEX_POINTS + "/_mvt/location/" + z + "/" + x + "/" + y);
-            mvtRequest.setJsonEntity("{\"size\" : 100, \"grid_type\": \"point\" }");
+            mvtRequest.setJsonEntity("{\"size\" : 100" + gridAgg + ",\"grid_type\": \"point\" }");
             final VectorTile.Tile tile = execute(mvtRequest);
             assertThat(tile.getLayersCount(), Matchers.equalTo(3));
             assertLayer(tile, HITS_LAYER, 4096, 33, 2);
@@ -330,7 +338,7 @@ public void testGridType() throws Exception {
         }
         {
             final Request mvtRequest = new Request(getHttpMethod(), INDEX_POINTS + "/_mvt/location/" + z + "/" + x + "/" + y);
-            mvtRequest.setJsonEntity("{\"size\" : 100, \"grid_type\": \"grid\" }");
+            mvtRequest.setJsonEntity("{\"size\" : 100" + gridAgg + ", \"grid_type\": \"grid\" }");
             final VectorTile.Tile tile = execute(mvtRequest);
             assertThat(tile.getLayersCount(), Matchers.equalTo(3));
             assertLayer(tile, HITS_LAYER, 4096, 33, 2);
@@ -340,7 +348,7 @@ public void testGridType() throws Exception {
         }
         {
             final Request mvtRequest = new Request(getHttpMethod(), INDEX_POINTS + "/_mvt/location/" + z + "/" + x + "/" + y);
-            mvtRequest.setJsonEntity("{\"size\" : 100, \"grid_type\": \"centroid\" }");
+            mvtRequest.setJsonEntity("{\"size\" : 100" + gridAgg + ", \"grid_type\": \"centroid\" }");
             final VectorTile.Tile tile = execute(mvtRequest);
             assertThat(tile.getLayersCount(), Matchers.equalTo(3));
             assertLayer(tile, HITS_LAYER, 4096, 33, 2);
@@ -354,6 +362,12 @@ public void testGridType() throws Exception {
             final ResponseException ex = expectThrows(ResponseException.class, () -> execute(mvtRequest));
             assertThat(ex.getResponse().getStatusLine().getStatusCode(), Matchers.equalTo(HttpStatus.SC_BAD_REQUEST));
         }
+        {
+            final Request mvtRequest = new Request(getHttpMethod(), INDEX_POINTS + "/_mvt/location/" + z + "/" + x + "/" + y);
+            mvtRequest.setJsonEntity("{\"grid_agg\": \"invalid_agg\" }");
+            final ResponseException ex = expectThrows(ResponseException.class, () -> execute(mvtRequest));
+            assertThat(ex.getResponse().getStatusLine().getStatusCode(), Matchers.equalTo(HttpStatus.SC_BAD_REQUEST));
+        }
     }
 
     public void testInvalidAggName() {

x-pack/plugin/vector-tile/src/main/java/org/elasticsearch/xpack/vectortile/feature/FeatureFactory.java

@@ -13,6 +13,8 @@
 import com.wdtinc.mapbox_vector_tile.adapt.jts.UserDataIgnoreConverter;
 import com.wdtinc.mapbox_vector_tile.build.MvtLayerProps;
 
+import org.elasticsearch.common.geo.GeoPoint;
+import org.elasticsearch.common.geo.SimpleFeatureFactory;
 import org.elasticsearch.common.geo.SphericalMercatorUtils;
 import org.elasticsearch.geometry.Circle;
 import org.elasticsearch.geometry.Geometry;
@@ -37,6 +39,7 @@
 import org.locationtech.jts.geom.TopologyException;
 import org.locationtech.jts.simplify.TopologyPreservingSimplifier;
 
+import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Comparator;
@@ -56,6 +59,8 @@ public class FeatureFactory {
     private final CoordinateSequenceFilter sequenceFilter;
     // pixel precision of the tile in the mercator projection.
     private final double pixelPrecision;
+    // optimization for points and rectangles
+    private final SimpleFeatureFactory simpleFeatureFactory;
 
     /**
      * The vector-tile feature factory will produce tiles as features based on the tile specification.
@@ -80,8 +85,33 @@ public FeatureFactory(int z, int x, int y, int extent, int padPixels) {
         this.builder = new JTSGeometryBuilder(geomFactory);
         this.clipTile = geomFactory.toGeometry(clipEnvelope);
         this.sequenceFilter = new MvtCoordinateSequenceFilter(tileEnvelope, extent);
+        this.simpleFeatureFactory = new SimpleFeatureFactory(z, x, y, extent);
     }
 
+    /**
+     * Returns a {@code byte[]} containing the mvt representation of the provided point
+     */
+    public byte[] point(double lon, double lat) throws IOException {
+        return simpleFeatureFactory.point(lon, lat);
+    }
+
+    /**
+     * Returns a {@code byte[]} containing the mvt representation of the provided rectangle
+     */
+    public byte[] box(double minLon, double maxLon, double minLat, double maxLat) throws IOException {
+        return simpleFeatureFactory.box(minLon, maxLon, minLat, maxLat);
+    }
+
+    /**
+     * Returns a {@code byte[]} containing the mvt representation of the provided points
+     */
+    public byte[] points(List<GeoPoint> multiPoint) {
+        return simpleFeatureFactory.points(multiPoint);
+    }
+
+    /**
+     * Returns a List {@code byte[]} containing the mvt representation of the provided geometry
+     */
     public List<byte[]> getFeatures(Geometry geometry) {
         // Get geometry in spherical mercator
         final org.locationtech.jts.geom.Geometry jtsGeometry = geometry.visit(builder);