Enrich Policies app

[cjcenizal]

API reference:

• Overview of enrichment
• PUT /_enrich/policy
• GET /_enrich/policy
• DELETE /_enrich/policy/{policy-name}
• PUT /_enrich/policy/{policy-name}/_execute
• GET /_enrich/_stats

Other references:

• ES enrichment PR with diagram explaining the process

Summary

Enrichment is the process by which documents are enriched with supplementary data drawn from a separate index (an "enrichment" index) during ingest time. Here's an example of how ingest node pipelines, enrichment indices, and enrich policies collaborate to ensure documents are enriched with up-to-date data:

1. A source index exists which contains a superset of enrichment data. In our example, we want to enrich documents with a user’s favorite food favoriteFood. We want to identify each user by their email, which is stored in the userEmail field.
2. We want to enrich ingested documents with the user's favorite food and identify them using their email, so we create a policy called "enrichWithFavoriteFood" that uses userEmail as the match field and contains favoriteFood in the enrich fields.
3. When we execute the "enrichWithFavoriteFood" policy, the enrich index is created with documents that contain these two fields.
4. We've configured an ingest node pipeline with an enrich processor with the "enrichWithFavoriteFood" enrich policy, a field of email, and a target field of favorites. When a document is ingested, this ingest node pipeline will use the value of the email field to find a document in the enrich index specified by the "enrichWithFavoriteFood" policy that has a userEmail field of the same value. Then it will copy the favoriteFood field of the enrich document onto the favorites field of the ingested document, resulting in a favorites.favoriteFood object.

Policy types

As of the writing of this issue, there are two policy types: match and geo_match. The backend engineers will notify us as they add new ones, and we'll need to update the UI to support them as they're added.

Execution and status

Enrich policies will be executed manually by the user. This means the UI will need to surface information on when the policy was last executed. @martijnvg will update the API to provide this information.

We'll also need to surface the current status of the enrich policy, i.e. whether it is currently executing and how long it's been executing for. We can use the Enrich Stats API to get information on whether a policy is executing or not.

List view UX

The home page of the app will be a list of the user's enrich policies. Here, the user will be able to perform "Delete" and "Execute" actions on each policy, and create new policies. Maybe we could drop the "Query" column if the table becomes too crowded.

👉 Note: The wireframe below shows that a "Type" is excluded, but this is out-of-date. There are two types we support, match and geo_match, so we will need a "Type" column to help people differentiate policies this way.

Creation form UX

👉 Note that good help text will be critical. Enrichment contains many moving parts and we'll need to name and explain things in a way that helps users build a mental model of what they're doing.

Step 1: Select data

The user defines the index from which to retrieve enrichment data. They can also specify a query if they want to limit the results.

Nice-to-have: A "Test" button which executes the ES request on the source index with the specified query, and shows you the results in a flyout.

Selecting indices

We could implement the index-selection UX in many ways:

• A simple option would be a text input that verifies the specified index exists.
• A more advanced option would be a typeahead which retrieves indices matching your search input, from which you can then select an index.
• The most advanced option would be a full-blown table that lets you search, sort, and select an index. Since index selection comes up in many of our UIs, we may want to build a robust reusable component for this.

👉Note: The wireframe below is outdated. We'll need to add a "Type" select dropdown so the user can choose between "Match" and "Geo match". We might want to consider using the EuiSuperSelect component so that we can populate each option with help text to explain the difference between the two.

Step 2: Select fields

The user selects the enrich fields and specifies the match field. We could try combining both interactions in a single table, or creating a different control for each. If we're pressed for time, we could reuse the field-selection components from the Rollup Job Wizard. It would be worth getting input from the design team here.

Information on which field is what type is in the table because I assume users may need this info select the correct enrich fields.

Step 3: Review and create

The user reviews the enrich policy configuration in a human-readable summary and can also preview the ES request that will be used to create the policy.

👉 We should add a "Create and execute" checkbox, similar to what we have in the Rollup Job Wizard. When checked, it will execute the policy as soon as the policy is created.

[elasticmachine]

Pinging @elastic/es-ui

[nicpenning]

Can we re-surface this Feature Request?

I think it would be create to have Enrich Policy management in the Kibana UI just like ILM, Ingest pipelines, etc...

I thought it would have been nice back then. Surprised this hasn't made it much further.

[alisonelizabeth]

Hi @nicpenning! Thanks for the ping. This is not currently on the team's roadmap, but we will update the issue if anything changes. Feel free to provide any additional information on why you think this feature should be prioritized and we will take it into consideration.

[nicpenning]

Thanks for the heads up!

I have been wanting this feature since day 1 that enrich policies were a possibility for us. Since we have many, it would be very handy to see all the enrich policies we have in the organization in a nice clean format instead of living in the Dev Tools console.

Also, it would make for helpful meta data adds to enrich policies such as when they were created, when they were updated/changed, and which ones are actively being used and where.

The growing use cases for enrich policies makes a UI useful since we would be able to not only manage but search and even create new policies as suggested above that we can test out on the the fly without needing to be an API wizard.

[sabarasaba]

We've added support for managing enrich policies within kibana with #164080