Skip to content

Commit 9db5942

Browse files
michaelcretzmanmichaelcretzman
authored
BYOTI content moved to Sec Overview Guide (#30158)
1 parent ffe6edf commit 9db5942

File tree

4 files changed

+137
-101
lines changed

4 files changed

+137
-101
lines changed

‎content/en/security/application_security/how-it-works/threat-intelligence.md

Lines changed: 2 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -39,107 +39,7 @@ To query for all traces containing threat intelligence from any source, use the
3939

4040
AAP supports enriching and searching traces with threat intelligence indicators of compromise stored in Datadog reference tables. [Reference Tables][2] allow you to combine metadata with information already in Datadog.
4141

42-
### Storing indicators of compromise in reference tables
43-
44-
Threat intelligence is supported in the CSV format and requires the following columns:
45-
46-
**CSV Structure**
47-
48-
| field | data | description| required | example|
49-
|------------------|-------|----|-----|--|
50-
| ip_address | text | The primary key for the reference table in the IPv4 dot notation format. | true | 192.0.2.1 |
51-
| additional_data | json | Additional data to enrich the trace. | false | `{"ref":"hxxp://example.org"}`
52-
| category | text | The threat intel [category][7]. This is used by some out of the box detection rules. | true | `residential_proxy` |
53-
| intention | text | The threat intel [intent][8]. This is used by some out of the box detection rules.| true | malicious | |
54-
| source | text | The name of the source and the link to its site, such as your team and your teams wiki. | true| `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | |
55-
56-
57-
58-
The full list of supported categories and intents is available at [Threat Intelligence Facets][3].
59-
60-
<div class="alert alert-info">JSON in a CSV requires double quoting. The following is an example CSV.</div>
61-
62-
```
63-
ip_address,additional_data,category,intention,source
64-
192.0.2.1,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
65-
192.0.2.2,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
66-
192.0.2.3,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
67-
```
68-
69-
### Uploading and enabling your own threat intel
70-
71-
Datadog supports creating reference tables through a manual upload, or by periodically retrieving the data from [Amazon S3, Azure storage, or Google Cloud storage][10].
72-
73-
Notes:
74-
- It can take 10 to 30 minutes to start enriching AAP traces after creating a table.
75-
- If a primary key is duplicated, it is skipped and an error message about the key is displayed.
76-
77-
On a new [references table][4] page:
78-
79-
1. Name the table. The table name is referenced in AAP's **Threat Intel** config.
80-
2. Upload a local CSV or import a CSV from a cloud storage bucket. The file is normalized and validated.
81-
3. Preview the table schema and choose the IP address as the Primary Key.
82-
83-
{{< img src="/security/application_security/threats/threat_intel/threat_intel_ref_table.png" alt="New reference table" style="width:100%;" >}}
84-
4. Save the table.
85-
5. In [Threat Intel][5], locate the new table, and then select the toggle to enable it.
86-
87-
<!-- {{< img src="/security/application_security/threats/threat_intel/threat_intel_ref_table_enabled.png" alt="Enabled reference table" style="width:100%;" >}} -->
88-
89-
#### Using cloud storage
90-
91-
When the reference table is created from cloud storage, it is refreshed periodically. The entire table is *replaced*. Data is not merged.
92-
93-
See the related reference table documentation for:
94-
- [Amazon S3][11]
95-
- [Azure storage][12]
96-
- [Google Cloud storage][13]
97-
98-
#### Troubleshooting cloud imports
99-
100-
If the reference tables are not refreshing, select the **View Change Events** link from the settings on the reference table detail page.
101-
102-
**View Change Events** opens a page in **Event Management** showing potential error events for the ingestion. You can also filter in **Event Management** using the reference table name.
103-
104-
<div class="alert alert-info">In Datadog Event Management, it can look like the data is fetched from the cloud, but it can take a few more minutes to propagate those changes to Threat Intellegence.</div>
105-
106-
Other useful cloud import details to remember:
107-
108-
- The expected latency before updated enrichments are available when a source is uploaded or updated is 10 to 30 minutes.
109-
- How to know when the updates are applied: The changes are visible in the reference table or in the spans. Select the **View Change Events** link from settings on the reference table detail page to see the related events.
110-
- The update replaces the *entire table* with the new data.
111-
- In case of a duplicated primary key, the rows with the duplicated key are not written, and an error is shown in the reference table detail page.
112-
113-
### Filter traces by joining the list with a Reference Table
114-
115-
You can filter AAP traces in Datadog by joining a trace table with a Reference Table.
116-
117-
To join a Reference Table with a trace query, you combine rows from the Datadog trace table and a Reference Table based on a related column between them. The traces query returns only those traces where there is a match in both tables.
118-
119-
Using a join with a Reference Table enables you to evaluate impact before enrichment by searching for historical matches with existing traces.
120-
121-
You can use any fields, not just IP addresses. For example, by associating security traces with specific URLs from a reference table, you can identify which parts of your application are being targeted by attacks. This can help pinpoint vulnerabilities or high-risk areas within the application.
122-
123-
Examples:
124-
125-
- Investigation and incident response. You can upload and join using IPs or other fields from attacks and see the traffic related to that incident.
126-
- By using security traces with the IP addresses from a Reference Table, such as associating IP addresses with geographic locations or organizational details, security teams can gain better context around attack attempts. This can help in understanding the origin and potential motivation behind the attacks.
127-
128-
129-
To join a trace with a Reference Table:
130-
131-
1. Upload the Reference Table you want to use as described in [Uploading and enabling your own threat intel](#uploading-and-enabling-your-own-threat-intel).
132-
2. To join a trace with a Reference Table, in [Traces][9], select **Add**, and then select **Join with Reference Table**.
133-
3. In **Inner join with reference table**, select the Reference Table to use.
134-
4. In **where field**, select the Datadog traces field to use for the join.
135-
5. In **column**, select the Reference Table field to use for the join.
136-
137-
<!-- {{< img src="security/application_security/threats/threat_intel/threat_intel_ref_join.png" alt="Your image description" style="width:100%;" >}} -->
138-
139-
### Enriching traces for detection rules
140-
141-
Enriching traces includes the threat intelligence attributes in AAP traces when the indicator of compromise matches the value of the `http.client_ip` key in the AAP trace. This enables searching for traces with threat intelligence matches using existing facets and using threat intelligence with detection rules.
142-
42+
For more information, see the [Bring Your Own Threat Intelligence][14] guide.
14343

14444

14545
## Threat intelligence in the user interface
@@ -165,3 +65,4 @@ Under `@threat_intel.results` you can always see the full details of what was ma
16565
[11]: /integrations/guide/reference-tables/?tab=amazons3#create-a-reference-table
16666
[12]: /integrations/guide/reference-tables/?tab=azurestorage#create-a-reference-table
16767
[13]: /integrations/guide/reference-tables/?tab=googlecloudstorage#create-a-reference-table
68+
[14]: /security/guide/byoti_guide

‎content/en/security/guide/_index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,5 @@ disable_toc: true
66

77
{{< whatsnext desc="General Guides:" >}}
88
{{< nextlink href="/security/guide/aws_fargate_config_guide" >}}AWS Fargate Configuration Guide for Datadog Security{{< /nextlink >}}
9+
{{< nextlink href="/security/guide/byoti_guide" >}}Bring Your Own Threat Intelligence{{< /nextlink >}}
910
{{< /whatsnext >}}

‎content/en/security/guide/byoti_guide.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
title: Bring Your Own Threat Intelligence
3+
disable_toc: false
4+
further_reading:
5+
- link: "logs/processing/pipelines"
6+
tag: "Documentation"
7+
text: "Log processing pipelines"
8+
---
9+
10+
Datadog Security supports enriching and searching [traces][14] with [threat intelligence][1] indicators of compromise stored in Datadog reference tables. [Reference Tables][2] allow you to combine metadata with information already in Datadog.
11+
12+
## Storing indicators of compromise in reference tables
13+
14+
Threat intelligence is supported in the CSV format and requires the following columns:
15+
16+
**CSV Structure**
17+
18+
| field | data | description| required | example|
19+
|------------------|-------|----|-----|--|
20+
| ip_address | text | The primary key for the reference table in the IPv4 dot notation format. | true | 192.0.2.1 |
21+
| additional_data | json | Additional data to enrich the trace. | false | `{"ref":"hxxp://example.org"}`
22+
| category | text | The threat intel [category][7]. This is used by some out of the box detection rules. | true | `residential_proxy` |
23+
| intention | text | The threat intel [intent][8]. This is used by some out of the box detection rules.| true | malicious | |
24+
| source | text | The name of the source and the link to its site, such as your team and your teams wiki. | true| `{"name":"internal_security_team", "url":"https://teamwiki.example.org"}` | |
25+
26+
27+
28+
The full list of supported categories and intents is available at [Threat Intelligence Facets][3].
29+
30+
<div class="alert alert-info">JSON in a CSV requires double quoting. The following is an example CSV.</div>
31+
32+
```
33+
ip_address,additional_data,category,intention,source
34+
192.0.2.1,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
35+
192.0.2.2,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
36+
192.0.2.3,"{""ref"":""hxxp://example.org""}",scanner,suspicious,"{""name"":""internal_security_team"", ""url"":""https://teamwiki.example.org""}"
37+
```
38+
39+
## Uploading and enabling your own threat intel
40+
41+
Datadog supports creating reference tables through a manual upload, or by periodically retrieving the data from [Amazon S3, Azure storage, or Google Cloud storage][10].
42+
43+
Notes:
44+
- It can take 10 to 30 minutes to start enriching AAP traces after creating a table.
45+
- If a primary key is duplicated, it is skipped and an error message about the key is displayed.
46+
47+
On a new [references table][4] page:
48+
49+
1. Name the table. The table name is referenced in AAP's **Threat Intel** config.
50+
2. Upload a local CSV or import a CSV from a cloud storage bucket. The file is normalized and validated.
51+
3. Preview the table schema and choose the IP address as the Primary Key.
52+
53+
{{< img src="/security/application_security/threats/threat_intel/threat_intel_ref_table.png" alt="New reference table" style="width:100%;" >}}
54+
4. Save the table.
55+
5. In [Threat Intel][5], locate the new table, and then select the toggle to enable it.
56+
57+
<!-- {{< img src="/security/application_security/threats/threat_intel/threat_intel_ref_table_enabled.png" alt="Enabled reference table" style="width:100%;" >}} -->
58+
59+
### Using cloud storage
60+
61+
When the reference table is created from cloud storage, it is refreshed periodically. The entire table is *replaced*. Data is not merged.
62+
63+
See the related reference table documentation for:
64+
- [Amazon S3][11]
65+
- [Azure storage][12]
66+
- [Google Cloud storage][13]
67+
68+
### Troubleshooting cloud imports
69+
70+
If the reference tables are not refreshing, select the **View Change Events** link from the settings on the reference table detail page.
71+
72+
**View Change Events** opens a page in **Event Management** showing potential error events for the ingestion. You can also filter in **Event Management** using the reference table name.
73+
74+
<div class="alert alert-info">In Datadog Event Management, it can look like the data is fetched from the cloud, but it can take a few more minutes to propagate those changes to Threat Intellegence.</div>
75+
76+
Other useful cloud import details to remember:
77+
78+
- The expected latency before updated enrichments are available when a source is uploaded or updated is 10 to 30 minutes.
79+
- How to know when the updates are applied: The changes are visible in the reference table or in the spans. Select the **View Change Events** link from settings on the reference table detail page to see the related events.
80+
- The update replaces the *entire table* with the new data.
81+
- In case of a duplicated primary key, the rows with the duplicated key are not written, and an error is shown in the reference table detail page.
82+
83+
## Filter traces by joining the list with a Reference Table
84+
85+
You can filter AAP traces in Datadog by joining a trace table with a Reference Table.
86+
87+
To join a Reference Table with a trace query, you combine rows from the Datadog trace table and a Reference Table based on a related column between them. The traces query returns only those traces where there is a match in both tables.
88+
89+
Using a join with a Reference Table enables you to evaluate impact before enrichment by searching for historical matches with existing traces.
90+
91+
You can use any fields, not just IP addresses. For example, by associating security traces with specific URLs from a reference table, you can identify which parts of your application are being targeted by attacks. This can help pinpoint vulnerabilities or high-risk areas within the application.
92+
93+
Examples:
94+
95+
- Investigation and incident response. You can upload and join using IPs or other fields from attacks and see the traffic related to that incident.
96+
- By using security traces with the IP addresses from a Reference Table, such as associating IP addresses with geographic locations or organizational details, security teams can gain better context around attack attempts. This can help in understanding the origin and potential motivation behind the attacks.
97+
98+
99+
To join a trace with a Reference Table:
100+
101+
1. Upload the Reference Table you want to use as described in [Uploading and enabling your own threat intel](#uploading-and-enabling-your-own-threat-intel).
102+
2. To join a trace with a Reference Table, in [Traces][9], select **Add**, and then select **Join with Reference Table**.
103+
3. In **Inner join with reference table**, select the Reference Table to use.
104+
4. In **where field**, select the Datadog traces field to use for the join.
105+
5. In **column**, select the Reference Table field to use for the join.
106+
107+
<!-- {{< img src="security/application_security/threats/threat_intel/threat_intel_ref_join.png" alt="Your image description" style="width:100%;" >}} -->
108+
109+
## Enriching traces for detection rules
110+
111+
Enriching traces includes the threat intelligence attributes in AAP traces when the indicator of compromise matches the value of the `http.client_ip` key in the AAP trace. This enables searching for traces with threat intelligence matches using existing facets and using threat intelligence with detection rules.
112+
113+
[1]: /security/threat_intelligence
114+
[2]: /integrations/guide/reference-tables
115+
[3]: /security/threat_intelligence/#threat-intelligence-facets
116+
[4]: https://app.datadoghq.com/reference-tables/create
117+
[5]: https://app.datadoghq.com/security/configuration/threat-intel
118+
[6]: https://app.datadoghq.com/security/configuration/asm/rules/edit/kdb-irk-nua?product=appsec
119+
[7]: /security/threat_intelligence#threat-intelligence-categories
120+
[8]: /security/threat_intelligence#threat-intelligence-intents
121+
[9]: https://app.datadoghq.com/security/appsec/traces
122+
[10]: /integrations/guide/reference-tables/?tab=manualupload#create-a-reference-table
123+
[11]: /integrations/guide/reference-tables/?tab=amazons3#create-a-reference-table
124+
[12]: /integrations/guide/reference-tables/?tab=azurestorage#create-a-reference-table
125+
[13]: /integrations/guide/reference-tables/?tab=googlecloudstorage#create-a-reference-table
126+
[14]: https://app.datadoghq.com/security/appsec/traces

‎content/en/security/threat_intelligence.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,12 @@ Threat Intelligence is reputation information that helps responders make informe
3030

3131
Datadog curates commercial, open-source, and in-house threat intelligence indicators of compromise into categories and intents. Threat intelligence is updated at least once per day, per source. This data is used to enrich your logs and traces with relevant reputation information.
3232

33+
## Bring your own threat intelligence
34+
35+
Datadog Security supports enriching and searching traces with threat intelligence indicators of compromise stored in Datadog reference tables. [Reference Tables][2] allow you to combine metadata with information already in Datadog.
36+
37+
For more information, see the [Bring Your Own Threat Intelligence][3] guide.
38+
3339
## Threat Intelligence Lifecycle
3440

3541
Datadog collects threat intelligence across the following entity types. Each entity type has unique characteristics and a useful timeframe. This timeframe, or lifecycle, requires consideration when assessing the importance of a threat intelligence match on your data.
@@ -115,3 +121,5 @@ Sources, categories, and intents are available as facets and filters on relevant
115121
{{< partial name="whats-next/whats-next.html" >}}
116122

117123
[1]:/security/detection_rules/
124+
[2]: /integrations/guide/reference-tables
125+
[3]: /security/guide/byoti_guide

0 commit comments

Comments
 (0)