diff --git a/docs/cse/administration/save-inventory-data-lookup-table.md b/docs/cse/administration/save-inventory-data-lookup-table.md
index add764f489..e5d612eb81 100644
--- a/docs/cse/administration/save-inventory-data-lookup-table.md
+++ b/docs/cse/administration/save-inventory-data-lookup-table.md
@@ -18,7 +18,7 @@ Although the Scheduled Search feature does support an **Alert Type** of “Save
## Prerequisites
-In order to create an inventory Lookup Table you need to have one or more sources of inventory data. One of the most common sources of inventory data in Sumo Logic is the Windows Active Directory Inventory source running on an Installed Collector. We recommend you collect AD logs every 12 hours, and that you do not collect logs more frequently than every 8 hours.
+In order to create an inventory Lookup Table, you need to have one or more sources of inventory data. One of the most common sources of inventory data in Sumo Logic is the [Windows Active Directory Inventory source](/docs/send-data/installed-collectors/sources/windows-active-directory-inventory-source/) running on an Installed Collector. We recommend you collect AD logs every 12 hours, and that you do not collect logs more frequently than every 8 hours.
Any inventory source–or any log source, for that matter–can be used to populate Lookup Tables. Sumo Logic also has a variety of inventory sources that run on Hosted Collectors, including the Okta and Carbon Black sources.
@@ -39,14 +39,14 @@ To create the Lookup Table schema:
1. Go to the Sumo Logic Library.
1. Navigate to the folder where you want to create the Lookup Table.
-1. Click **Add New** and then select **New Lookup**.
})
-1. The **Create Lookup Table** page appears.
})
+1. Click **Add New** and then select **New Lookup**.
+1. The **Create Lookup Table** page appears.
1. **Name**. Enter a name for the Lookup Table.
1. **Description**. (Optional)
1. **Set a TTL (Time to Live) for table entries**? Click **No**.
1. **Choose a size limit handling option**. This option controls how additions to the Lookup table will be handled when it reaches its size limit (100 MB). Click **Delete Old Data.**
1. **Create Lookup Table** Click **Create Schema only**.
-1. The page displays a **Schema** section. (The screenshot below shows the schema settings for our example filled in.)
})
+1. The page displays a **Schema** section. (The screenshot below shows the schema settings for our example filled in.)
1. For the first column, enter:
* **Fields**. Enter *mail*.
* **Value Type**. Leave the default, *string*, selected.
@@ -75,7 +75,7 @@ Where:
* `_collector` identifies the collector where the Active Directory source runs.
* `PATH` is the path of the lookup table, in this format: `path://"/Library/Admin Recommended/userIdToUsername"` You can copy the path to the Lookup Table in the Sumo Logic Library. Hover over the row for the table in the Library, and select **Copy path to clipboard** from the three-dot kebab menu.
- })
+
## Step 3: Save and schedule the search
@@ -85,13 +85,13 @@ Be sure to choose “Email” as the **Alert type**. (*Don’t* select **Save to
To save and schedule the search:
-1. In the log search tab where you’ve run your query, choose **Save as** from the three-dot kebab menu in the query area.
})
+1. In the log search tab where you’ve run your query, choose **Save as** from the three-dot kebab menu in the query area.
1. On the **Save Item** popup:
* **Name**. Enter a name for the query.
* **Time range**. Select a time range for the query.
* **Search By**. Select *Receipt Time*.
* **Location to save to**. Choose a folder location.
- * Click **Schedule this search**.
})
+ * Click **Schedule this search**.
1. On the **Save Item** popup:
* **Run frequency**. Select *Daily*, unless you have another preference.
* **Send Notification**. Choose *If the following condition is met*.
@@ -99,7 +99,7 @@ To save and schedule the search:
* **Alert type**. Select *Email*.
* **Number of results**. Enter *5*, or another value if you prefer.
* **Recipients.** Enter the email addresses of one or more users to receive email alerts.
- * **Include in email**. Select *Search Query* and *Histogram*, unless you have another preference.
})
+ * **Include in email**. Select *Search Query* and *Histogram*, unless you have another preference.
1. Click **Save.**
## Step 4: Configure the Lookup Table in Cloud SIEM
diff --git a/docs/cse/records-signals-entities-insights/configure-entity-lookup-table.md b/docs/cse/records-signals-entities-insights/configure-entity-lookup-table.md
index 8ae0cdc2e6..82fb7de2e1 100644
--- a/docs/cse/records-signals-entities-insights/configure-entity-lookup-table.md
+++ b/docs/cse/records-signals-entities-insights/configure-entity-lookup-table.md
@@ -26,7 +26,16 @@ In addition, in some systems a user or a host has both a name and a unique ID, t
* `d8ece0f8-10a4-3c62-b8a3-2e636a3a0509`
* `testk-122.testlabs.local`
-Multiple identifiers for the same user or host are a problem when it comes to correlating Signals around a common Entity: unless you allow for all permutations of a username or hostname, your rule or search won’t function as intended with all data sources.
+Multiple identifiers for the same user or host are a problem when it comes to correlating Signals around a common Entity. Unless you allow for all permutations of a username or hostname, your rule or search won’t function as intended with all data sources.
+
+### Examples of when you create Lookup Tables
+
+Following are some examples of situations when you'd want to use Entity Lookup Tables:
+* CrowdStrike FDR data uses an agent ID (AID) instead of a hostname for some messages.
+* Mail Transfer Agent (MTA) systems report usernames in an email format.
+* Your users have different login names on different systems (for example, Windows, Linux, and AWS).
+
+### How does an Entity Lookup Table work?
An Entity Lookup Table defines two sets of values: a lookup value to look for in an incoming message and a substitution value. You can create Entity Lookup Tables to support the following types of normalization:
@@ -34,17 +43,17 @@ An Entity Lookup Table defines two sets of values: a lookup value to look for in
* **User ID to Normalized Username**
* **Username to Normalized Username**
-Entity Lookup Tables are based on Sumo Logic’s Lookup Tables feature. Here is an example of a **Host ID to Normalized Hostname** Lookup Table in the Sumo Logic Library:
+Entity Lookup Tables are based on Sumo Logic’s [Lookup Tables](/docs/search/lookup-tables/) feature. Here is an example of a **Host ID to Normalized Hostname** Lookup Table in the Sumo Logic Library:
})
-## Limitations
+## Creating a Lookup Table
-You can configure a maximum of five Entity Lookup Tables.
+Before you configure a Lookup Table in Cloud SIEM, you must [create the Lookup Table](/docs/search/lookup-tables/create-lookup-table/) in the Sumo Logic platform. There are a variety of ways to create a Lookup Table.
-## Creating a Lookup Table
+### Limitations
-Before you configure a Lookup Table in Cloud SIEM, you must create the Lookup Table in the Sumo Logic platform. There are a variety of ways to create a Lookup Table.
+You can configure a maximum of five Entity Lookup Tables.
### Populate table from inventory data
@@ -56,11 +65,11 @@ This method–the typical way to populate a Lookup Table for the purpose of Enti
If you already have a Lookup Table that contains normalization data, you can configure it in Cloud SIEM. Or, if you have existing normalization data that is not currently in a Lookup Table you can create a Lookup Table with that data. Note that your Lookup Table must contain a field that contains a lookup value and one that contains a substitution value. There is no requirement for particular column names.
-For instructions, see the Create a Lookup Table topic. After creating the table, perform the steps in [Configure the Lookup Table in Cloud SIEM](#configure-the-lookup-table-in-cloud-siem), below.
+For instructions, see [Create a Lookup Table](/docs/search/lookup-tables/create-lookup-table/). After creating the table, perform the steps in [Configure the Lookup Table in Cloud SIEM](#configure-the-lookup-table-in-cloud-siem), below.
### Configure the Lookup Table in Cloud SIEM
-After you’ve created your Entity Lookup Table in the Sumo Logic Library, you can configure it in Cloud SIEM.
+After you've [created your Entity Lookup Table](/docs/search/lookup-tables/create-lookup-table/) in the Sumo Logic Library, you can configure it in Cloud SIEM.
1. [**Classic UI**](/docs/cse/introduction-to-cloud-siem/#classic-ui). In the top menu select **Configuration**, and then under **Entities** select **Normalization**.
[**New UI**](/docs/cse/introduction-to-cloud-siem/#new-ui). In the top menu select **Configuration**, and then under **Cloud SIEM Entities** select **Normalization**. You can also click the **Go To...** menu at the top of the screen and select **Normalization**.
1. On the **Entity Normalization** page, click **Lookup Tables**.
diff --git a/docs/cse/schema/record-processing-pipeline.md b/docs/cse/schema/record-processing-pipeline.md
index 14f7625e27..55ea92afdb 100644
--- a/docs/cse/schema/record-processing-pipeline.md
+++ b/docs/cse/schema/record-processing-pipeline.md
@@ -32,7 +32,7 @@ The key-value pairs are input to the next step of the process: mapping.
## Map message fields to schema attributes
-The mapping process creates a Record from the key-value pairs that were extracted from a message, and maps a subset of the keys to Cloud SIEM schema attributes.
+The [mapping process](/docs/cse/schema/create-structured-log-mapping/) creates a Record from the key-value pairs that were extracted from a message, and maps a subset of the keys to Cloud SIEM schema attributes.
Mapping solves a particular problem: messages from different products use different names to identify users, applications, devices and so on. For example, some messages may refer to a source IP address as `sourceIP`, while others use `sourceIpAddress`. We need a standard set of names for the data that most messages are likely to contain. The [Cloud SIEM schema](/docs/cse/schema) defines that standard set of names.
@@ -40,7 +40,7 @@ What’s the benefit of mapping? It results in Records that use a common (standa
## Normalize usernames and hostnames
-Username and hostname normalization is the process of transforming the value of Record attributes that contain usernames and hostnames into a standard format. The normalized value replaces the non-normalized value in a Record. The non-normalized values of hostname and usernames are retained in a `_raw` field in the Record.
+[Username and hostname normalization](/docs/cse/schema/username-and-hostname-normalization/) is the process of transforming the value of Record attributes that contain usernames and hostnames into a standard format. The normalized value replaces the non-normalized value in a Record. The non-normalized values of hostname and usernames are retained in a `_raw` field in the Record.
Why normalize? Assume Cloud SIEM receives messages with an email-type field "bob@bobsbaitshop.com" and username-type field "bob". We can use normalization to transform "bob@bobsbaitshop.com" to "bob", allowing the username and email to be correlated together.
diff --git a/docs/cse/schema/username-and-hostname-normalization.md b/docs/cse/schema/username-and-hostname-normalization.md
index 67e27e529c..f33a786e1c 100644
--- a/docs/cse/schema/username-and-hostname-normalization.md
+++ b/docs/cse/schema/username-and-hostname-normalization.md
@@ -7,14 +7,18 @@ description: Learn about how Cloud SIEM normalizes usernames and hostnames durin
import useBaseUrl from '@docusaurus/useBaseUrl';
-This topic describes how Cloud SIEM normalizes usernames and hostnames in Records during the parsing and mapping process. This allows for common name forms among Active Directory, AWS, and fully qualified domain names to be normalized into a domain and username form.
+Cloud SIEM normalizes usernames and hostnames in Records during the parsing and mapping process. This allows for common name forms among Active Directory, AWS, and fully qualified domain names to be normalized into a domain and username form.
-## Details
+## Getting data into Cloud SIEM for normalization
-The name normalization process normalizes a complete username into a base username and a domain. The following forms of usernames are
-normalized.
+Data that is normalized comes into Cloud SIEM from several sources:
+* **Active Directory**. Configure the [Windows Active Directory Inventory Source](/docs/send-data/installed-collectors/sources/windows-active-directory-inventory-source/). The username, hostname, and domain information is pulled in directly from Active Directory to Cloud SIEM for normalization. For information on the computer and user data that is obtained, see the [Windows Active Directory Inventory Source](/docs/cse/administration/inventory-sources-and-data/#windows-active-directory-inventory-source) section in [Inventory Sources and Data](/docs/cse/administration/inventory-sources-and-data).
+* **AWS**. Configure AWS sources and ensure that the **Forward to SIEM** option is set to true. For more information, see [Cloud SIEM Ingestion Best Practices](/docs/cse/ingestion/cse-ingestion-best-practices/) and [Ingestion Sources for Cloud SIEM](/docs/cse/ingestion/ingestion-sources-for-cloud-siem/).
+* **Other sources**. See [Inventory Sources and Data](/docs/cse/administration/inventory-sources-and-data).
-Some of the common forms for username are:
+## Normalization process
+
+The name normalization process normalizes a complete username into a base username and a domain. Some of the common forms for username are:
* `username` (no domain)
* `AD-DOMAIN\username`
@@ -42,25 +46,37 @@ The following fields of the schema are normalized.
When a username is normalized, the original, un-normalized name is placed in a `_raw` name attribute, for example, `user_useraname_raw`. The normalized name is placed in the attribute field `user_username`. The rules engine allows the `_raw` username forms to be used in rule creation.
-It’s important to note, that if no name normalization configuration exists, the name attribute will consist of the original (non-normalized) form and the system will continue to operate as it does today, with the exception that that `_raw` attribute will also be populated.
+If a name normalization configuration exists, the name attribute will be populated with the form `:` where the `` portion is not populated for the normalized default domain. When name normalization is enabled all name fields (not-raw) will be lowercase. For more information, see [Single domain example](#single-domain-example) and [Multiple domains example](#multiple-domains-example) below.
+
+:::note
+If no name normalization configuration exists, the name attribute will consist of the original (non-normalized) form and the system will continue to operate as it does today, with the exception that that `_raw` attribute will also be populated.
+:::
-If a name normalization configuration exists, the name attribute will be populated with the form `:` where the `` portion is not populated for the normalized default domain. When name normalization is enabled all name fields (not-raw) will be lowercase. For more information, see [Example - single Domain](#example---single-domain) and [Example - multiple domains](#example---multiple-domains), below.
+## Configure entity normalization
-## Configuration
+1. [**Classic UI**](/docs/cse/introduction-to-cloud-siem/#classic-ui). In the top menu select **Configuration**, and then under **Entities** select **Normalization**.
[**New UI**](/docs/cse/introduction-to-cloud-siem/#new-ui). In the top menu select **Configuration**, and then under **Cloud SIEM Entities** select **Normalization**. You can also click the **Go To...** menu at the top of the screen and select **Normalization**.
+1. Select the **Domain** tab. (For information about the **Lookup Tables** tab, see [Configure an Entity Lookup Table](/docs/cse/records-signals-entities-insights/configure-entity-lookup-table/)).
+1. You can configure just **Username Normalization**, just **Hostname Normalization**, or both. We recommend you enable both.
+1. Under **Normalization Formats** there are configuration options to normalize names from:
+ * **FQDN**. Normalize names in the form `user@somedomain.net` or `hostname.somedomain.net`.
+ * **Active Directory**. Normalize active directory domains username and hostname formats.
+ * **AWS**. Normalize AWS ARN and usernames.
+1. In **Normalized Default Domain**, provide a default domain name.
+1. In **Domain Name Mappings**, enter mappings for secondary domains.
+1. Click **Save**.
-The name normalization feature can be enabled in the **Incoming Data** section under the **Entities** tab of the Cloud SIEM configuration.
+Following is an example configuration:
})
-You can configure just username normalization, just hostname normalization, or both. We recommend you enable both.
+### Warnings and issues
-There are configuration options to normalize names (“Normalization Formats”) from:
+If no name normalization is configured, the system will continue to operate as it does today. If normalization is then enabled, any signals already created in the system will use the non-normalized form of the name. Any new signals will use the normalized name. This means there is potential for insights to be uncorrelated between the two different name forms for one insight window. This is especially true as all usernames will now be lowercase.
-* FQDN - Normalize names in the form `user@somedomain.net` or `hostname.somedomain.net`
-* Active Directory - Normalize active directory domains username and hostname formats
-* AWS - Normalize AWS ARN and Usernames
-## Default domains
+## Domain normalization
+
+### Default domains
When normalization is configured, at least one domain must be configured and a “Normalized Default Domain” must be provided. The default name will never show up in normalized names, as it’s assumed, and username forms with no domain portion will be considered part of that domain. In our example above, we’ve assumed the name “sumo”.
@@ -73,7 +89,7 @@ Next, the user should enter the domain name forms that will be seen in the custo
These domains should all have the “Normalized Name” that matches the “Normalized Default Domain”, for example, `sumo`.
-## Secondary domains
+### Secondary domains
The normalization configurations also supports secondary domains that may not map users in a different namespace to the same name. For example, if `bob@sumologic.com` is not the same as `bob@jask.com`, then a secondary domain should be configured. In this case a second set of configurations should be populated to maps to a different “normalized domain” (`jask`).
@@ -84,19 +100,9 @@ For example, in this case a configuration could be:
In this case, these domains map to a different normalized domain (`jask`). When one of these domains is normalized, it will show up as `bob:jask` in the normalized name form.
-## Warnings and issues
-
-If no name normalization is configured, the system will continue to operate as it does today. If normalization is then enabled, any signals already created in the system will use the non-normalized form of the name. Any new signals will use the normalized name. This means there is potential for insights to be uncorrelated between the two different name forms for one insight window. This is especially true as all usernames will now be lowercase.
-
-## Example UI
-
-An example UI is provided for a case where the customer has a domain name `test.com` and an active directory domain named `test`.
-
-})
-
-### Example - single domain
+### Single domain example
-In this example, it is assumed the user has configured the system for “Primary domain” and has configured the domains `SUMO` and `sumologic.com`. In this case, assume a log line has a username field:
+In this example, it is assumed you have configured the system for “Primary domain” and you configured the domains `SUMO` and `sumologic.com`. In this case, assume a log line has a username field:
`bob`
@@ -143,9 +149,9 @@ The normalized username would be:
`user_username_raw = JASK\fred`
-#### Example - multiple domains
+### Multiple domains example
-In this example, it is assumed the user has configured the system for “Primary domain” and has also introduced a sub-domain (JASK). In this case, the configuration looks like:
+In this example, it is assumed you have configured the system for “Primary domain” and also introduced a sub-domain (JASK). In this case, the configuration looks like:
Normalized Default Domain: sumo
@@ -166,4 +172,11 @@ Name forms matching the default domain would look like:
| `fred@jask.com` | `fred:jask` |
| `JASK\fred` | `fred:jask` |
| `bob@someothername.com` | `bob@someothername.com` |
-| `OTHERDOMAIN\suzy` | `otherdomain\suzy` |
+| `OTHERDOMAIN\suzy` | `otherdomain\suzy` |
+
+### Active Directory domain example
+
+Following is an example configuration for a case where the customer has a domain name `test.com` and an Active Directory domain named `test`.
+
+
+
diff --git a/docs/search/lookup-tables/create-lookup-table.md b/docs/search/lookup-tables/create-lookup-table.md
index 971acf44be..db818f6c31 100644
--- a/docs/search/lookup-tables/create-lookup-table.md
+++ b/docs/search/lookup-tables/create-lookup-table.md
@@ -6,18 +6,21 @@ description: Learn about lookup tables and how to create and manage them.
import useBaseUrl from '@docusaurus/useBaseUrl';
-This section has instructions for creating and and managing Lookup Tables using the Sumo Logic UI.
+This section has instructions for creating Lookup Tables using the Sumo Logic UI.
-:::tip
-You can also use the [Lookups API](https://api.sumologic.com/docs/#tag/lookupManagement) to create and manage Lookup Tables.
-:::
-
-For information about updating, exporting, and sharing Lookup Tables, see [Manage and Update Lookup Tables](manage-update-lookup-tables.md).
+For additional articles about lookup tables, see the following:
+* To update, export, and share Lookup Tables, see [Manage and Update Lookup Tables](manage-update-lookup-tables.md).
+* To configure a Lookup Table for use in Cloud SIEM, see [Configure an Entity Lookup Table](/docs/cse/records-signals-entities-insights/configure-entity-lookup-table/).
+* To populate a Lookup Table with Cloud SIEM inventory data, see [Save Inventory Data to a Lookup Table](/docs/cse/administration/save-inventory-data-lookup-table/).
:::note
New Lookup Tables are available in all deployments except Sumo Logic's Montreal deployment, pending AWS providing a required AWS service in the Montreal region.
:::
+:::tip
+You can also use the [Lookups API](https://api.sumologic.com/docs/#tag/lookupManagement) to create and manage Lookup Tables.
+:::
+
## Introduction to Lookup Tables
A Lookup Table is a table of data hosted on Sumo Logic that you can use to enrich the log data received by Sumo Logic. For example, in a Sumo Logic log search, you'd refer to a Lookup Table of user account data to map the user ID in an incoming log to a row in the Lookup Table, and return other attributes of that user, for instance, email address or phone number. The fields you look up appear as part of your search results.
diff --git a/static/img/cse/Configuration.png b/static/img/cse/Configuration.png
index fe34e96893..783131d1dc 100644
Binary files a/static/img/cse/Configuration.png and b/static/img/cse/Configuration.png differ
diff --git a/static/img/cse/Example_UI.png b/static/img/cse/Example_UI.png
index cf13912cce..1fb77c48ee 100644
Binary files a/static/img/cse/Example_UI.png and b/static/img/cse/Example_UI.png differ