Add support for geo_shape fields as the entity geospatial field when …

…creating tracking containment alerts (#164100) Closes #163996 ### To test 1) Checkout [fake tracks geo_shape branch](https://github.com/nreese/faketracks/tree/geo_shape) 2) run npm install 3) run `node ./generate_tracks.js` 4) in kibana, create `tracks*` data view 5) create map, use "create index" and draw boundaries that intersect tracks. See screen shot <img width="500" alt="Screen Shot 2023-08-17 at 2 49 52 PM" src="https://github.com/elastic/kibana/assets/373691/5f1444d7-2e12-4dd2-99c1-c730c2157e04"> 6) create geo containment alert where entity index is `tracks*` and boundaries index is `boundaries`. 7) Verify alerts get generated with entity geo_shape locations --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
elastic · Aug 23, 2023 · 3393d87 · 3393d87
1 parent e651a6f
commit 3393d87
Show file tree

Hide file tree

Showing 16 changed files with 148 additions and 91 deletions.
diff --git a/docs/user/alerting/rule-types/geo-rule-types.asciidoc b/docs/user/alerting/rule-types/geo-rule-types.asciidoc
@@ -10,7 +10,7 @@ In the event that an entity is contained within a boundary, an alert may be gene
 === Requirements
 To create a tracking containment rule, the following requirements must be present:
 
-- *Tracks index or data view*: An index containing a `geo_point` field, `date` field,
+- *Tracks index or data view*: An index containing a `geo_point` or `geo_shape` field, `date` field,
 and some form of entity identifier. An entity identifier is a `keyword` or `number`
 field that consistently identifies the entity to be tracked. The data in this index should be dynamically
 updating so that there are entity movements to alert upon.
@@ -36,12 +36,7 @@ as well as two Kuery bars used to provide additional filtering context for each
 image::user/alerting/images/alert-types-tracking-containment-conditions.png[Define the condition to detect,width=75%]
 // NOTE: This is an autogenerated screenshot. Do not edit it directly.
 
-Index (entity):: This clause requires an *index or data view*, a *time field* that will be used for the *time window*, and a *`geo_point` field* for tracking.
-When entity:: This clause specifies which crossing option to track. The values
-*Entered*, *Exited*, and *Crossed* can be selected to indicate which crossing conditions
-should trigger a rule. *Entered* alerts on entry into a boundary, *Exited* alerts on exit
-from a boundary, and *Crossed* alerts on all boundary crossings whether they be entrances
-or exits.
+Index (entity):: This clause requires an *index or data view*, a *time field* that will be used for the *time window*, and a *`geo_point` or `geo_shape` field* for tracking.
 Index (Boundary):: This clause requires an *index or data view*, a *`geo_shape` field*
 identifying boundaries, and an optional *Human-readable boundary name* for better alerting
 messages.

diff --git a/...ntainment/query_builder/__snapshots__/geo_containment_alert_type_expression.test.tsx.snap b/...ntainment/query_builder/__snapshots__/geo_containment_alert_type_expression.test.tsx.snap
diff --git a/...s/public/rule_types/geo_containment/query_builder/expressions/entity_index_expression.tsx b/...s/public/rule_types/geo_containment/query_builder/expressions/entity_index_expression.tsx
@@ -161,7 +161,7 @@ export const EntityIndexExpression: FunctionComponent<Props> = ({
       isInvalid={isInvalid}
       value={indexPattern.title}
       defaultValue={i18n.translate('xpack.stackAlerts.geoContainment.entityIndexSelect', {
-        defaultMessage: 'Select a data view and geo point field',
+        defaultMessage: 'Select a data view and geospatial field',
       })}
       popoverContent={indexPopover}
       expressionDescription={i18n.translate('xpack.stackAlerts.geoContainment.entityIndexLabel', {

diff --git a/x-pack/plugins/stack_alerts/public/rule_types/geo_containment/readme.md b/x-pack/plugins/stack_alerts/public/rule_types/geo_containment/readme.md
@@ -4,7 +4,7 @@ There are several steps required to set up geo containment alerts for testing in
 that allows you to view triggered alerts as they happen. These instructions outline
 how to load test data, but really these steps can be used to load any data for geo
 containment alerts so long as you have the following data:
-- An index containing a`geo_point` field and a `date` field. This data is presumed to
+- An index containing a`geo_point` or `geo_shape` field and a `date` field. This data is presumed to
 be dynamic (updated).
 - An index containing `geo_shape` data, such as boundary data, bounding box data, etc.
 This data is presumed to be static (not updated). Shape data matching the query is

diff --git a/x-pack/plugins/stack_alerts/public/rule_types/geo_containment/types.ts b/x-pack/plugins/stack_alerts/public/rule_types/geo_containment/types.ts
@@ -23,6 +23,5 @@ export interface GeoContainmentAlertParams extends RuleTypeParams {
   boundaryIndexQuery?: Query;
 }
 
-// Will eventually include 'geo_shape'
-export const ES_GEO_FIELD_TYPES = ['geo_point'];
+export const ES_GEO_FIELD_TYPES = ['geo_point', 'geo_shape'];
 export const ES_GEO_SHAPE_TYPES = ['geo_shape'];
diff --git a/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/executor.test.ts b/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/executor.test.ts
@@ -220,15 +220,17 @@ describe('getGeoContainmentExecutor', () => {
         {
           dateInShape: '2021-04-28T16:56:11.923Z',
           docId: 'ZVBoGXkBsFLYN2Tj1wmV',
-          location: [-73.99018926545978, 40.751759740523994],
+          location: [0, 0],
+          locationWkt: 'POINT (-73.99018926545978 40.751759740523994)',
           shapeLocationId: 'kFATGXkBsFLYN2Tj6AAk',
         },
       ],
       '1': [
         {
           dateInShape: '2021-04-28T16:56:11.923Z',
           docId: 'ZlBoGXkBsFLYN2Tj1wmV',
-          location: [-73.99561604484916, 40.75449890457094],
+          location: [0, 0],
+          locationWkt: 'POINT (-73.99561604484916 40.75449890457094)',
           shapeLocationId: 'kFATGXkBsFLYN2Tj6AAk',
         },
       ],

diff --git a/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/alert_context.test.ts b/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/alert_context.test.ts
@@ -9,6 +9,31 @@ import { getContainedAlertContext, getRecoveredAlertContext } from './alert_cont
 import { OTHER_CATEGORY } from '../constants';
 
 test('getContainedAlertContext', () => {
+  expect(
+    getContainedAlertContext({
+      entityName: 'entity1',
+      containment: {
+        location: [0, 0],
+        locationWkt: 'POINT (100 0)',
+        shapeLocationId: 'boundary1Id',
+        dateInShape: '2022-06-21T16:56:11.923Z',
+        docId: 'docId',
+      },
+      shapesIdsNamesMap: { boundary1Id: 'boundary1Name' },
+      windowEnd: new Date('2022-06-21T17:00:00.000Z'),
+    })
+  ).toEqual({
+    containingBoundaryId: 'boundary1Id',
+    containingBoundaryName: 'boundary1Name',
+    detectionDateTime: '2022-06-21T17:00:00.000Z',
+    entityDateTime: '2022-06-21T16:56:11.923Z',
+    entityDocumentId: 'docId',
+    entityId: 'entity1',
+    entityLocation: 'POINT (100 0)',
+  });
+});
+
+test('getContainedAlertContext for backwards compatible number[] location format', () => {
   expect(
     getContainedAlertContext({
       entityName: 'entity1',
@@ -37,7 +62,8 @@ describe('getRecoveredAlertContext', () => {
     const activeEntities = new Map();
     activeEntities.set('entity1', [
       {
-        location: [100, 0],
+        location: [0, 0],
+        locationWkt: 'POINT (100 0)',
         shapeLocationId: 'boundary1Id',
         dateInShape: '2022-06-21T16:56:11.923Z',
         docId: 'docId',
@@ -63,7 +89,8 @@ describe('getRecoveredAlertContext', () => {
     const inactiveEntities = new Map();
     inactiveEntities.set('entity1', [
       {
-        location: [100, 0],
+        location: [0, 0],
+        locationWkt: 'POINT (100 0)',
         shapeLocationId: OTHER_CATEGORY,
         dateInShape: '2022-06-21T16:56:11.923Z',
         docId: 'docId',

diff --git a/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/alert_context.ts b/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/alert_context.ts
@@ -50,7 +50,10 @@ function getAlertContext({
     entityId: entityName,
     entityDateTime: containment.dateInShape || null,
     entityDocumentId: containment.docId,
-    entityLocation: `POINT (${containment.location[0]} ${containment.location[1]})`,
+    entityLocation:
+      containment.locationWkt !== undefined
+        ? containment.locationWkt
+        : `POINT (${containment.location[0]} ${containment.location[1]})`,
     detectionDateTime: new Date(windowEnd).toISOString(),
   };
   if (!isRecovered) {

diff --git a/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/es_query_builder.ts b/x-pack/plugins/stack_alerts/server/rule_types/geo_containment/lib/es_query_builder.ts
@@ -63,13 +63,16 @@ export async function executeEsQuery(
                         },
                       },
                     ],
-                    docvalue_fields: [
+                    fields: [
                       entity,
                       {
                         field: dateField,
                         format: 'strict_date_optional_time',
                       },
-                      geoField,
+                      {
+                        field: geoField,
+                        format: 'wkt',
+                      },
                     ],
                     _source: false,
                   },

diff --git a/...ack_alerts/server/rule_types/geo_containment/lib/get_entities_and_generate_alerts.test.ts b/...ack_alerts/server/rule_types/geo_containment/lib/get_entities_and_generate_alerts.test.ts
@@ -54,6 +54,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
       [
         {
           location: [0, 0],
+          locationWkt: 'POINT (0 0)',
           shapeLocationId: '123',
           dateInShape: 'Wed Dec 09 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
           docId: 'docId1',
@@ -65,6 +66,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
       [
         {
           location: [0, 0],
+          locationWkt: 'POINT (0 0)',
           shapeLocationId: '456',
           dateInShape: 'Wed Dec 16 2020 15:31:31 GMT-0700 (Mountain Standard Time)',
           docId: 'docId2',
@@ -76,6 +78,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
       [
         {
           location: [0, 0],
+          locationWkt: 'POINT (0 0)',
           shapeLocationId: '789',
           dateInShape: 'Wed Dec 23 2020 16:31:31 GMT-0700 (Mountain Standard Time)',
           docId: 'docId3',
@@ -141,6 +144,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
         [
           {
             location: [0, 0],
+            locationWkt: 'POINT (0 0)',
             shapeLocationId: '999',
             dateInShape: 'Wed Dec 09 2020 12:31:31 GMT-0700 (Mountain Standard Time)',
             docId: 'docId7',
@@ -166,6 +170,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
         [
           {
             location: [0, 0],
+            locationWkt: 'POINT (0 0)',
             shapeLocationId: '999',
             dateInShape: 'Wed Dec 09 2020 12:31:31 GMT-0700 (Mountain Standard Time)',
             docId: 'docId7',
@@ -204,6 +209,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
     const currLocationMapWithOther = new Map([...currLocationMap]).set('d', [
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: OTHER_CATEGORY,
         dateInShape: 'Wed Dec 09 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
@@ -225,6 +231,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
           [
             {
               location: [0, 0],
+              locationWkt: 'POINT (0 0)',
               shapeLocationId: 'other',
               dateInShape: 'Wed Dec 09 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
               docId: 'docId1',
@@ -241,18 +248,21 @@ describe('getEntitiesAndGenerateAlerts', () => {
     const currLocationMapWithThreeMore = new Map([...currLocationMap]).set('d', [
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '789',
         dateInShape: 'Wed Dec 10 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '123',
         dateInShape: 'Wed Dec 08 2020 12:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId2',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '456',
         dateInShape: 'Wed Dec 07 2020 10:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId3',
@@ -277,18 +287,21 @@ describe('getEntitiesAndGenerateAlerts', () => {
     const currLocationMapWithOther = new Map([...currLocationMap]).set('d', [
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: OTHER_CATEGORY,
         dateInShape: 'Wed Dec 10 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '123',
         dateInShape: 'Wed Dec 08 2020 12:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '456',
         dateInShape: 'Wed Dec 07 2020 10:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
@@ -310,18 +323,21 @@ describe('getEntitiesAndGenerateAlerts', () => {
     const currLocationMapWithOther = new Map([...currLocationMap]).set('d', [
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '123',
         dateInShape: 'Wed Dec 10 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: OTHER_CATEGORY,
         dateInShape: 'Wed Dec 08 2020 12:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
       },
       {
         location: [0, 0],
+        locationWkt: 'POINT (0 0)',
         shapeLocationId: '456',
         dateInShape: 'Wed Dec 07 2020 10:31:31 GMT-0700 (Mountain Standard Time)',
         docId: 'docId1',
@@ -338,6 +354,7 @@ describe('getEntitiesAndGenerateAlerts', () => {
       new Map([...currLocationMap]).set('d', [
         {
           location: [0, 0],
+          locationWkt: 'POINT (0 0)',
           shapeLocationId: '123',
           dateInShape: 'Wed Dec 10 2020 14:31:31 GMT-0700 (Mountain Standard Time)',
           docId: 'docId1',

diff --git a/...ns/stack_alerts/server/rule_types/geo_containment/lib/get_entities_and_generate_alerts.ts b/...ns/stack_alerts/server/rule_types/geo_containment/lib/get_entities_and_generate_alerts.ts
@@ -56,6 +56,11 @@ export function getEntitiesAndGenerateAlerts(
       return;
     }
 
+    // TODO remove otherCatIndex check
+    // Elasticsearch filters aggregation is used to group results into buckets matching entity locations intersecting boundary shapes
+    // filters.other_bucket_key returns bucket with entities that did not intersect any boundary shape.
+    // shapeLocationId === OTHER_CATEGORY can only occur when containments.length === 1
+    // test data does not follow this pattern and needs to be updated.
     const otherCatIndex = containments.findIndex(
       ({ shapeLocationId }) => shapeLocationId === OTHER_CATEGORY
     );