Skip to content

Commit

Permalink
Merge branch 'develop' into pr/13265
Browse files Browse the repository at this point in the history
  • Loading branch information
rhetoric101 committed May 30, 2023
2 parents fe0a223 + 8c2746b commit 54ec279
Show file tree
Hide file tree
Showing 59 changed files with 976 additions and 388 deletions.
17 changes: 14 additions & 3 deletions scripts/verify_mdx.js
Original file line number Diff line number Diff line change
@@ -1,16 +1,27 @@
const frontmatter = require('@github-docs/frontmatter');
const mdx = require('@mdx-js/mdx');
const fs = require('fs');
const glob = require('glob');

const readFile = async (filePath) => {
let failed = false;
console.log(`reading ${filePath}`);

const mdxText = fs.readFileSync(filePath, 'utf8');
try {
console.log(`reading ${filePath}`);
const mdxText = fs.readFileSync(filePath, 'utf8');
const jsx = mdx.sync(mdxText);
} catch (exception) {
console.log(JSON.stringify(exception, null, 4));
return filePath;
failed = true;
}

const { errors } = frontmatter(mdxText);
if (errors.length > 0) {
console.log(JSON.stringify(errors, null, 4));
failed = true;
}

return failed ? filePath : null;
};

const main = async () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Here's a brief summary of the capabilities of each user type:
<tbody>
<tr>
<td>
Basic users are free. Basic users can set up our observability tools, run queries of your data, use custom <InlinePopover type="dashboards" />, use some basic alerting features, and more. They **can't** use our curated experiences (for example, our <InlinePopover type="apm" /> UI, browser UI, or mobile UI).
Basic users are free. Basic users can set up our observability tools, run queries on your data, use custom <InlinePopover type="dashboards" /> (quickstart dashboards up to 7 days), use some basic alerting features, and more. Basic users **can't** use our curated experiences (for example, our <InlinePopover type="apm" /> UI, browser UI, or mobile UI).

</td>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ If called outside of the context of a monitored web request or background task,
`expected`

_boolean_, _iterable[String]_, _callable(exception_class, exception_instance, traceback)->boolean_
</td>_
</td>

<td>
Optional. Errors to mark as expected can be passed in as an iterable of strings in the form `module:class`. This value can also be a callable or a Boolean indicating whether the error is expected. These errors will be reported to the UI but will not affect Apdex score or error rate.
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/journey-app-slow/external-services.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ By the end of this tutorial, you will:
title="Screenshot showing the opening map for external services"
alt="Screenshot showing the opening map for external services"
src={apmExternalServiceIntro}
style={{ maxWidth: '20%' }}
style={{ maxWidth: '50%' }}
/>
</Side>
</SideBySide>
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/journey-app-slow/root-causes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Use the time picker to look for spikes over various time ranges.
title="Time picker"
alt="Image showing the time picker for APM"
src={journeyTime}
style={{ maxWidth: '20%' }}
style={{ maxWidth: '60%' }}
/>
</Callout>

Expand Down
145 changes: 145 additions & 0 deletions src/content/docs/tutorial-large-logs/filter-large-logs.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
title: Reduce log complexity and cost by filtering
metaDescription: "Reduce the complexity and cost of your log management by filtering your logs with drop rules"
---

import logsIngestPipeline from 'images/logs_diagram_ingest-pipeline.webp'

import logsDropFilterSelection1 from 'images/logs_screenshot-full_drop-filter-selection-1.webp'

import logsAttribute from 'images/logs_screenshot-full_drop-filter-attribute.webp'

import logsDrop from 'images/logs_screenshot-drop-log.webp'

import logsIntro from 'images/logs_screenshot-full_intro.webp'


Modern systems create massive amounts of logs. Not all of those are useful. In fact, there's a high chance when you look at your logs you'd find *most* aren't useful. You might have a service that spews logs for every page load or a backup service which you never need to monitor logs for.


<SideBySide>
<Side>

With New Relic you can create drop rules that look at your logs and ignore logs that you haven't selected for ingest. This has a few key benefits:


* Lower costs by storing only the logs relevant to your account.
* Lower costs by dropping specific attributes
* Reduce toil by only storing valuable logs.

</Side>
<Side>

<img
title="Logs architecture for drop filters in New Relic"
alt="Diagram of logs architecture for drop filters in New Relic"
src={logsIntro}
/>

</Side>
</SideBySide>


## How drop filter rules work [#how-it-works]

A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to [the New Relic database (NRDB)](/docs/data-apis/get-started/nrdb-horsepower-under-hood/).

This creates an distinction between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it can't be queried: the data is gone and cannot be restored.

## Decide which logs to drop [#decide]

Deciding which logs to keep and which logs to drop is a highly specific decision for each team and organization. Logs valuable to one organization may not be valuable to another. Regardless, here are a few suggestions on how to decide which logs are valuable and which to drop:

* **What logs does your team rely on today?**: If your team already manually reviews a subset of logs regularly, that indicates those logs are valuable and should not be dropped. Likewise, if there is a set of logs your team never looks at that might indicate they should be dropped.
* **What apps and systems produce the most logs?**: An app or system that creates a large amount of logs indicates you should spend time deciding what to do with those logs. Is it a valuable and widely used app which indicates you should keep most of the logs? Is it a redundent system which is spewing logs with minimal value?

Do take note that while an app or system may be rarely used, that doesn't mean its logs have no value. You would hate to drop logs from an application that is barely used only for that application to go down in a few months with no easy way to troubleshoot.

<img
title="Logs architecture for drop filters in New Relic"
alt="Diagram of logs architecture for drop filters in New Relic"
src={logsIngestPipeline}
/>

<figcaption>
During ingest, customer log data can be parsed, transformed, or dropped before being stored in the New Relic database (NRDB).
</figcaption>

## Filter your log ingest [#filter-steps]

The following steps will guide you through how to drop logs in the New Relic UI.


Let's say Acme Corp creates 2TB of logs each day. They decide this is too many logs to ingest for both cost and usability reasons. They take a look at their logs and realize over half of their daily logs are from a legacy Node.js application. While they need to keep the app around, they don't care for the logs created by this app. They decided to drop all logs created from the Node.js app.


<Steps>
<Step>
### Navigate to the UI

Go to **[one.newrelic.com > Logs](https://one.newrelic.com/launcher/logger.log-launcher)**
</Step>
<Step>
### Create your drop rule

Filter or query to the specific set of logs that contain the data you want to drop.

There are a few ways to do this, but the easiest is to query for the logs you want to drop. In this case, you would do the following:


<SideBySide>
<Side>

1. Select **All partitions** near the search bar.
2. Enter their query. In this case `logtype=node`.
3. Press enter and confirm the correct logs appear.
4. Once the query is active, click **Create drop filter** on the left nav.
5. Give the drop rule a meaningful name.
6. Save the drop filter rule.
</Side>
<Side>
<img
title="Query log "
alt="An image showing how to query for the logs you will drop"
src={logsDrop}
/>
</Side>
</SideBySide>

</Step>

<Step>
### Drop attributes

Acme Corp still wants to reduce their ingest. They decided that they don't need certain attributes in their stored logs, so they decide to drop attributes such as `purchase_order`.

<SideBySide>
<Side>

1. In **All logs** click on a log that contains the attribute you want to drop to open log detail view.
2. Click on the attribute you want to drop to open the attribute menu. In this case `purchase_order`.
3. Click **Create drop filter from attribute**.
4. Give the drop rule a meaningful name.
5. Save the drop filter rule.
</Side>
<Side>
<img
title="Query attribute"
alt="An image showing how to query for the attributes you will drop"
src={logsAttribute}
/>
</Side>
</SideBySide>
</Step>
</Steps>

Repeat the above steps as many times as required until you're happy with your log ingest. If you need help querying for logs and attributes, [check out our doc on log specific syntax](/docs/logs/ui-data/query-syntax-logs/) or our doc on [more complex log filtering](/docs/logs/ui-data/drop-data-drop-filter-rules/).

<DocTiles numbered>
<DocTile title='Get started' path="/docs/journey-large-logs/get-started" ></DocTile>
<DocTile title='Filter and reduce your log ingest' label={{text: 'You are here', color: '#FCD672'}} path="/docs/journey-large-logs/filter" ></DocTile>
</DocTiles>

<DocTiles>
<DocTile title='Organize your logs' number='3'path="/docs/journey-large-logs/organize" ></DocTile>
</DocTiles>

0 comments on commit 54ec279

Please sign in to comment.