Add Archive Search documentation #32032
Open
+175
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Preview links (active after the
|
git-thuerk-done
requested changes
Oct 7, 2025
| This feature supports logs archived through: | ||
|
|
||
| - [Datadog Log Management archives][1] | ||
| - [Observability Pipelines archives][2] |
There was a problem hiding this comment.
Suggested change
| - [Observability Pipelines archives][2] | |
| - [Observability Pipelines archives][2] |
this links to https://docs.datadoghq.com/observability_pipelines/destinations/amazon_s3/, just want to ensure this is correct ?
Co-authored-by: Alicia Thuerk <26307719+git-thuerk-done@users.noreply.github.com>
git-thuerk-done
approved these changes
Oct 7, 2025
There was a problem hiding this comment.
Thanks for the updates, looks good! I had one last comment on this link but approved :)
#32032 (comment)
|
|
||
| ### Typical use cases | ||
|
|
||
| Archive Search is ideal when you need to query logs that are stored but not indexed. |
There was a problem hiding this comment.
Should we say "Stored in External Archive" to make it clearer ?
There was a problem hiding this comment.
Suggested change
| Archive Search is ideal when you need to query logs that are stored but not indexed. | |
| Archive Search is ideal when you need to query logs that are stored in an external archive but not indexed. |
|
|
||
| To improve query performance and reduce scan volume: | ||
| - Narrow the time range and use selective filters. | ||
| - Filter on **indexed attributes** such as `transaction_id` or `user_id`. |
There was a problem hiding this comment.
this will not be available day 1 of the preview
(configuring indexed attribute and optimize search against it) - can we remove it for now? should come end of the quarter
There was a problem hiding this comment.
Suggested change
| - Filter on **indexed attributes** such as `transaction_id` or `user_id`. |
|
|
||
| ### Adding role delegation to S3 archives | ||
|
|
||
| Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. |
There was a problem hiding this comment.
can we rename to "searching"?
There was a problem hiding this comment.
Suggested change
| Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. | |
| Datadog only supports searching from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. |
|
|
||
| ## Cloud-specific permissions | ||
|
|
||
| Datadog requires the permission to read from your archives to rehydrate content from them. This permission can be changed at any time. |
There was a problem hiding this comment.
rename to "searching"?
estherk15
commented
Oct 8, 2025
|
|
||
| ### Typical use cases | ||
|
|
||
| Archive Search is ideal when you need to query logs that are stored but not indexed. |
There was a problem hiding this comment.
Suggested change
| Archive Search is ideal when you need to query logs that are stored but not indexed. | |
| Archive Search is ideal when you need to query logs that are stored in an external archive but not indexed. |
| From the **Archive Search list view**, you can: | ||
|
|
||
| - **Stop** a running search: preserves logs already retrieved. | ||
| - **Delete** a completed search: removes results and metadata. |
There was a problem hiding this comment.
Suggested change
| - **Delete** a completed search: removes results and metadata. |
| - **Delete** a completed search: removes results and metadata. | ||
| - **Duplicate** a search: opens the Archive Search creation form with the same parameters for efficient reruns. | ||
|
|
||
| {{< img src="logs/archive_search/list_view.png" alt="Archive Search list view showing stop, delete, and duplicate options" style="width:90%;" >}} |
There was a problem hiding this comment.
Need to update this image since delete is not going to be available as an option
Suggested change
| {{< img src="logs/archive_search/list_view.png" alt="Archive Search list view showing stop, delete, and duplicate options" style="width:90%;" >}} |
|
|
||
| To improve query performance and reduce scan volume: | ||
| - Narrow the time range and use selective filters. | ||
| - Filter on **indexed attributes** such as `transaction_id` or `user_id`. |
There was a problem hiding this comment.
Suggested change
| - Filter on **indexed attributes** such as `transaction_id` or `user_id`. |
|
|
||
| ## Cloud-specific permissions | ||
|
|
||
| Datadog requires the permission to read from your archives to rehydrate content from them. This permission can be changed at any time. |
There was a problem hiding this comment.
Suggested change
| Datadog requires the permission to read from your archives to rehydrate content from them. This permission can be changed at any time. | |
| Datadog requires the permission to read your archives to search content from them. This permission can be changed at any time. |
|
|
||
| ### Adding role delegation to S3 archives | ||
|
|
||
| Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. |
There was a problem hiding this comment.
Suggested change
| Datadog only supports rehydrating from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. | |
| Datadog only supports searching from archives that have been configured to use role delegation to grant access. After you have modified your Datadog IAM role to include the IAM policy above, ensure that each archive in your [archive configuration page][3] has the correct AWS Account + Role combination. |
|
|
||
| {{% tab "Azure Storage" %}} | ||
|
|
||
| Datadog uses an Azure AD group with the Storage Blob Data Contributor role scoped to your archives' storage account to rehydrate log events. You can grant this role to your Datadog service account from your storage account's Access Control (IAM) page by [assigning the Storage Blob Data Contributor role to your Datadog integration app][1]. |
There was a problem hiding this comment.
Suggested change
| Datadog uses an Azure AD group with the Storage Blob Data Contributor role scoped to your archives' storage account to rehydrate log events. You can grant this role to your Datadog service account from your storage account's Access Control (IAM) page by [assigning the Storage Blob Data Contributor role to your Datadog integration app][1]. | |
| Datadog uses an Azure AD group with the Storage Blob Data Contributor role scoped to your archives' storage account to search log events. You can grant this role to your Datadog service account from your storage account's Access Control (IAM) page by [assigning the Storage Blob Data Contributor role to your Datadog integration app][1]. |
|
|
||
| {{% tab "Google Cloud Storage" %}} | ||
|
|
||
| In order to rehydrate log events from your archives, Datadog uses a service account with the Storage Object Viewer role. You can grant this role to your Datadog service account from the [Google Cloud IAM Admin page][1] by editing the service account's permissions, adding another role, and then selecting **Storage > Storage Object Viewer**. |
There was a problem hiding this comment.
Suggested change
| In order to rehydrate log events from your archives, Datadog uses a service account with the Storage Object Viewer role. You can grant this role to your Datadog service account from the [Google Cloud IAM Admin page][1] by editing the service account's permissions, adding another role, and then selecting **Storage > Storage Object Viewer**. | |
| In order to search log events from your archives, Datadog uses a service account with the Storage Object Viewer role. You can grant this role to your Datadog service account from the [Google Cloud IAM Admin page][1] by editing the service account's permissions, adding another role, and then selecting **Storage > Storage Object Viewer**. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Merge instructions
Merge readiness:
Do not merge Pending PM approval