-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'develop' into pr/13265
- Loading branch information
Showing
59 changed files
with
976 additions
and
388 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
145 changes: 145 additions & 0 deletions
145
src/content/docs/tutorial-large-logs/filter-large-logs.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
--- | ||
title: Reduce log complexity and cost by filtering | ||
metaDescription: "Reduce the complexity and cost of your log management by filtering your logs with drop rules" | ||
--- | ||
|
||
import logsIngestPipeline from 'images/logs_diagram_ingest-pipeline.webp' | ||
|
||
import logsDropFilterSelection1 from 'images/logs_screenshot-full_drop-filter-selection-1.webp' | ||
|
||
import logsAttribute from 'images/logs_screenshot-full_drop-filter-attribute.webp' | ||
|
||
import logsDrop from 'images/logs_screenshot-drop-log.webp' | ||
|
||
import logsIntro from 'images/logs_screenshot-full_intro.webp' | ||
|
||
|
||
Modern systems create massive amounts of logs. Not all of those are useful. In fact, there's a high chance when you look at your logs you'd find *most* aren't useful. You might have a service that spews logs for every page load or a backup service which you never need to monitor logs for. | ||
|
||
|
||
<SideBySide> | ||
<Side> | ||
|
||
With New Relic you can create drop rules that look at your logs and ignore logs that you haven't selected for ingest. This has a few key benefits: | ||
|
||
|
||
* Lower costs by storing only the logs relevant to your account. | ||
* Lower costs by dropping specific attributes | ||
* Reduce toil by only storing valuable logs. | ||
|
||
</Side> | ||
<Side> | ||
|
||
<img | ||
title="Logs architecture for drop filters in New Relic" | ||
alt="Diagram of logs architecture for drop filters in New Relic" | ||
src={logsIntro} | ||
/> | ||
|
||
</Side> | ||
</SideBySide> | ||
|
||
|
||
## How drop filter rules work [#how-it-works] | ||
|
||
A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to [the New Relic database (NRDB)](/docs/data-apis/get-started/nrdb-horsepower-under-hood/). | ||
|
||
This creates an distinction between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it can't be queried: the data is gone and cannot be restored. | ||
|
||
## Decide which logs to drop [#decide] | ||
|
||
Deciding which logs to keep and which logs to drop is a highly specific decision for each team and organization. Logs valuable to one organization may not be valuable to another. Regardless, here are a few suggestions on how to decide which logs are valuable and which to drop: | ||
|
||
* **What logs does your team rely on today?**: If your team already manually reviews a subset of logs regularly, that indicates those logs are valuable and should not be dropped. Likewise, if there is a set of logs your team never looks at that might indicate they should be dropped. | ||
* **What apps and systems produce the most logs?**: An app or system that creates a large amount of logs indicates you should spend time deciding what to do with those logs. Is it a valuable and widely used app which indicates you should keep most of the logs? Is it a redundent system which is spewing logs with minimal value? | ||
|
||
Do take note that while an app or system may be rarely used, that doesn't mean its logs have no value. You would hate to drop logs from an application that is barely used only for that application to go down in a few months with no easy way to troubleshoot. | ||
|
||
<img | ||
title="Logs architecture for drop filters in New Relic" | ||
alt="Diagram of logs architecture for drop filters in New Relic" | ||
src={logsIngestPipeline} | ||
/> | ||
|
||
<figcaption> | ||
During ingest, customer log data can be parsed, transformed, or dropped before being stored in the New Relic database (NRDB). | ||
</figcaption> | ||
|
||
## Filter your log ingest [#filter-steps] | ||
|
||
The following steps will guide you through how to drop logs in the New Relic UI. | ||
|
||
|
||
Let's say Acme Corp creates 2TB of logs each day. They decide this is too many logs to ingest for both cost and usability reasons. They take a look at their logs and realize over half of their daily logs are from a legacy Node.js application. While they need to keep the app around, they don't care for the logs created by this app. They decided to drop all logs created from the Node.js app. | ||
|
||
|
||
<Steps> | ||
<Step> | ||
### Navigate to the UI | ||
|
||
Go to **[one.newrelic.com > Logs](https://one.newrelic.com/launcher/logger.log-launcher)** | ||
</Step> | ||
<Step> | ||
### Create your drop rule | ||
|
||
Filter or query to the specific set of logs that contain the data you want to drop. | ||
|
||
There are a few ways to do this, but the easiest is to query for the logs you want to drop. In this case, you would do the following: | ||
|
||
|
||
<SideBySide> | ||
<Side> | ||
|
||
1. Select **All partitions** near the search bar. | ||
2. Enter their query. In this case `logtype=node`. | ||
3. Press enter and confirm the correct logs appear. | ||
4. Once the query is active, click **Create drop filter** on the left nav. | ||
5. Give the drop rule a meaningful name. | ||
6. Save the drop filter rule. | ||
</Side> | ||
<Side> | ||
<img | ||
title="Query log " | ||
alt="An image showing how to query for the logs you will drop" | ||
src={logsDrop} | ||
/> | ||
</Side> | ||
</SideBySide> | ||
|
||
</Step> | ||
|
||
<Step> | ||
### Drop attributes | ||
|
||
Acme Corp still wants to reduce their ingest. They decided that they don't need certain attributes in their stored logs, so they decide to drop attributes such as `purchase_order`. | ||
|
||
<SideBySide> | ||
<Side> | ||
|
||
1. In **All logs** click on a log that contains the attribute you want to drop to open log detail view. | ||
2. Click on the attribute you want to drop to open the attribute menu. In this case `purchase_order`. | ||
3. Click **Create drop filter from attribute**. | ||
4. Give the drop rule a meaningful name. | ||
5. Save the drop filter rule. | ||
</Side> | ||
<Side> | ||
<img | ||
title="Query attribute" | ||
alt="An image showing how to query for the attributes you will drop" | ||
src={logsAttribute} | ||
/> | ||
</Side> | ||
</SideBySide> | ||
</Step> | ||
</Steps> | ||
|
||
Repeat the above steps as many times as required until you're happy with your log ingest. If you need help querying for logs and attributes, [check out our doc on log specific syntax](/docs/logs/ui-data/query-syntax-logs/) or our doc on [more complex log filtering](/docs/logs/ui-data/drop-data-drop-filter-rules/). | ||
|
||
<DocTiles numbered> | ||
<DocTile title='Get started' path="/docs/journey-large-logs/get-started" ></DocTile> | ||
<DocTile title='Filter and reduce your log ingest' label={{text: 'You are here', color: '#FCD672'}} path="/docs/journey-large-logs/filter" ></DocTile> | ||
</DocTiles> | ||
|
||
<DocTiles> | ||
<DocTile title='Organize your logs' number='3'path="/docs/journey-large-logs/organize" ></DocTile> | ||
</DocTiles> |
Oops, something went wrong.