Skip to content

Commit

Permalink
feat(sonic): Create manage large number of logs tutorial
Browse files Browse the repository at this point in the history
  • Loading branch information
@bradleycamacho
bradleycamacho committed May 25, 2023
1 parent d6dc336 commit a809f54
Show file tree
Hide file tree
Showing 10 changed files with 532 additions and 0 deletions.
145 changes: 145 additions & 0 deletions src/content/docs/journey-large-logs/filter.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
title: Filter log ingest
metaDescription: "Test"
---

import logsIngestPipeline from 'images/logs_diagram_ingest-pipeline.webp'

import logsDropFilterSelection1 from 'images/logs_screenshot-full_drop-filter-selection-1.webp'

import logsAttribute from 'images/logs_screenshot-full_drop-filter-attribute.webp'

import logsDrop from 'images/logs_screenshot-drop-log.webp'

import logsIntro from 'images/logs_screenshot-full_intro.webp'


As we've talked about before, modern systems create massive amounts of logs. Not all of those are useful. In fact, there's a high chance when you look at your logs you'd find *most* aren't useful.


<SideBySide>
<Side>

New Relic provides the ability to create rule sets that look at your logs and ignore logs that you haven't selected for ingest. This has a few key benefits:


* Lower costs by storing only the logs relevant to your account.
* Lower costs by dropping specific attributes
* Reduce toil by only storing valuable logs.

</Side>
<Side>

<img
title="Logs architecture for drop filters in New Relic"
alt="Diagram of logs architecture for drop filters in New Relic"
src={logsIntro}
/>

</Side>
</SideBySide>


## How drop filter rules work [#how-it-works]

A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to the New Relic database (NRDB).

This creates an explicit demarcation between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it cannot be queried: the data is gone and cannot be restored.

## Decide which logs to drop [#decide]

Deciding which logs to keep and which logs to drop is a highly specific decision for each team and organization. Logs valuable to one organization may not be valuable to another. Regardless, here are a few suggestions on how to decide which logs are valuable and which to drop:

* **What logs do your team already use**: If your team already manually reviews a subset of logs regularly, that indicates those logs are valuable and should not be dropped. Likewise, if there is a set of lgos your team never looks at that might indicate they should be dropped.
* **What apps and systems are producing the most logs**: an app or system that creates a large amount of logs indicates you should spend time deciding what to do with those logs. Is it a valuable and widely used app which indicates you should keep most of the logs? Is it a redundent system which is spewing logs with minimal value?

Do take note that while an app or system may be rarely used, that doesn't mean logs are always not valuable. You would hate to drop logs from an application that is barely used only for that application to go down in a few months with no easy way to troubleshoot.

<img
title="Logs architecture for drop filters in New Relic"
alt="Diagram of logs architecture for drop filters in New Relic"
src={logsIngestPipeline}
/>

<figcaption>
During the ingestion process, customer log data can be parsed, transformed, or dropped before being stored in the New Relic database (NRDB).
</figcaption>

## Filter your log ingest [#filter-steps]

The following steps will guide you through how to drop logs in the New Relic UI.


Let's say Acme Corp creates 2TB of logs each day. They decide this too many logs to ingest for both cost and usability reasons. They take a look at their logs and realize over half of their daily logs are from a legacy Node.js application. While they need to keep the app around, they don't care for the logs created by this app. They decided to drop all logs created from the Node app.


<Steps>
<Step>
### Navigate to the UI

Go to **[one.newrelic.com > Logs](https://one.newrelic.com/launcher/logger.log-launcher)**
</Step>
<Step>
### Create your drop rule

Filter or query to the specific set of logs that contain the data to be dropped.

There are a few ways to do this, but the easiest is to query for the logs you want to drop. In this case, ACME Corp would do the following:


<SideBySide>
<Side>

1. Select **All partitions** near the search bar
2. Enter their query. In this case `logtype=node`
3. Press enter and confirm the correct logs appear.
4. Once the query is active, from **Manage data** on the left nav of the Logs UI, click **Create drop filter**.
5. Give the drop rule a meaningful name.
6. Save the drop filter rule.
</Side>
<Side>
<img
title="Query log "
alt="An image showing how to query for the logs you will drop"
src={logsDrop}
/>
</Side>
</SideBySide>

</Step>

<Step>
### Drop attributes

ACME Corp still wants to reduce their ingest. They decided that they don't need certain attributes in their stored logs, so they decide to drop attributes such as `purchase_order`.

<SideBySide>
<Side>

1. In **All logs** click on a log that contains the attribute you want to drop to open log detail view.
2. Click on the attribute you want to drop to open the attribute menu. In this case `purchase_order`.
3. Click **Create drop filter from attribute**.
4. Give the drop rule a meaningful name.
5. Save the drop filter rule.
</Side>
<Side>
<img
title="Query attribute"
alt="An image showing how to query for the attributes you will drop"
src={logsAttribute}
/>
</Side>
</SideBySide>
</Step>
</Steps>

Repeate the above steps as many times as required until you're happy with your log ingest. If you need help querying for logs and attributes, [check out our doc on log specific syntax](/docs/logs/ui-data/query-syntax-logs/).

<DocTiles numbered>
<DocTile title='Get started' path="/docs/journey-large-logs/get-started" ></DocTile>
<DocTile title='Filter and reduce your log ingest' label={{text: 'You are here', color: '#FCD672'}} path="/docs/journey-large-logs/filter" ></DocTile>
</DocTiles>

<DocTiles>
<DocTile title='Organize your logs' number='3'path="/docs/journey-large-logs/organize" ></DocTile>
</DocTiles>
178 changes: 178 additions & 0 deletions src/content/docs/journey-large-logs/get-started.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
---
title: How to manage large log ingest
metaDescription: "Test"
---

import logsParsing from 'images/logs_screenshot_full-parsing.webp'
import logsPartition from 'images/logs_screenshot_full-partition.webp'


Modern systems create large amounts of log data. You might be dealing with hundreds of gigabytes to dozens of terabytes today, and the amount will continue to increase as your system scales. Anything past a few gigabytes of logs means hours of toil while you search through logs when need arises. Log management solutions, like New Relic, provide the tools to handle large sets of logs and make them managable — and more important, valuable.

This tutorial walks you through how to us New Relic to manage a large amount of log ingest. You'll start by forwarding your logs to New Relic, which means sending your log data to New Relic automatically. You'll then identify what logs to ingest and which to drop. Finally you'll organize your logs through partitions and parsing.


<img
title="log-dashboard"
alt="An image displaying New Relic's log monitoring dashboard"
src={journeyLogsDash}
/>

## Choosing a log management platform [#choose]


Once you've identified you have a problem with managing logs, it's time to choose a log management platform. There are many platforms out there. Some focus on quick automation but sacrifice ease-of-use. Others focus on complex features, but obscure their pricing.

New Relic's philosphy when it comes to log management focuses on three things: we want out logs solution to be **flexible, transparent, and usage based**. Let's quickly talk about what these mean:

* **Flexible**: everyone needs different things from their logs. Some may need to ingest a large amount for record keeping while some may need to ingest a small amount. Some may need to heavily parse their logs while other may barely parse their logs at all. New Relic provides a platform flexible enough to meet everyone's needs.
* **Transparent and usage-based**: only pay for logs you ingest. Not all logs are valuable, so there's no use in ingesting and paying for logs you will never use. In this tutorial we'll explore how to selectively ingest logs in an affordable and effective manner.

## Let's begin: forward your logs [#forward]


To forward your log data to New Relic, choose one or more of these options:

<table>
<thead>
<tr>
<th>
Log forwarding option
</th>
<th>
When to use
</th>
<th>
Install
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
APM agent
</td>
<td>
By default, our APM agents do two things: add metadata to your logs, which gives you logs in context (ability to see logs data in various relevant places in our platform) and forward your logs to New Relic.

This is a popular option for DevOps teams and smaller organizations because it lets you easily report application logs, with no additional third-party solutions required. [Learn more about APM logs.](/docs/apm/new-relic-apm/getting-started/get-started-logs-context)
</td>
<td>
<SideBySide>
<Side>
<TechTileGrid>

<TechTile
name="Go agent"
icon="logo-go"
to="https://docs.newrelic.com/docs/apm/agents/go-agent/configuration/go-agent-logging/"
/>

<TechTile
name="Java agent"
icon="logo-java"
to="https://one.newrelic.com/marketplace?account=&state=bc1aa4bb-2cbb-cce1-db7e-df9debdf4b27"
/>

<TechTile
name=".NET agent"
icon="logo-dotnet"
to="https://one.newrelic.com/marketplace?account=&state=97a39162-d880-5fd3-a5f0-330bb44595fa"
/>

<TechTile
name="Node.js agent"
icon="logo-nodejs"
to="https://one.newrelic.com/marketplace/install-data-source?account=&state=c4521520-1485-7926-fc41-3f12bf575964"
/>
</TechTileGrid>
</Side>
<Side>
<TechTileGrid>
<TechTile
name="PHP agent"
icon="logo-php"
to="https://one.newrelic.com/nr1-core?state=aa633b41-72d4-009c-3abf-55dcf64894fe"
/>

<TechTile
name="Python agent"
icon="logo-python"
to="https://one.newrelic.com/nr1-core?state=20fda75b-58fb-a92a-f9e1-7b052035c6e8"
/>

<TechTile
name="Ruby agent"
icon="logo-ruby"
to="https://one.newrelic.com/nr1-core?state=d69143ab-605c-579b-25bf-cc6e5fee5b80"
/>

</TechTileGrid>
</Side>
</SideBySide>



</td>
</tr>
<tr>
<td>
Infrastructure agent
</td>
<td>
With our infrastructure agent, you can capture any logs present on your host, including your app logs.

Compared to using an APM agent to report logs, this can take a little more setting up but gives you much more powerful options (for example, ability to collect custom attributes, which you can't do with <InlinePopover type="apm" /> agents).
</td>
<td>
[Install the infrastructure agent](/docs/logs/forward-logs/forward-your-logs-using-infrastructure-agent/)
</td>
</tr>
<tr>
<td>
Third-party log services
</td>
<td>
We have [a wide range of integrations for other log services](/docs/logs/forward-logs/enable-log-management-new-relic), including Amazon, Microsoft, Fluentd, Fluent Bit, Kubernetes, Logstash, and more.
</td>
<td>
[Third-party log solutions](https://one.newrelic.com/marketplace?account=1606862&state=2ae57d9a-eb5e-d8ab-c5aa-31c9c00bceb7)
</td>
</tr>
<tr>
<td>
Log API or TCP endpoint
</td>
<td>
When you want more precision control about what and how logs are sent to New Relic.
</td>
<td>
[Log API](/docs/logs/log-api/introduction-log-api/) or [TCP endpoint](/docs/logs/log-api/use-tcp-endpoint-forward-logs-new-relic)
</td>
</tr>
<tr>
<td>
OpenTelemetry SDK
</td>
<td>
Forward logs from your apps to an OpenTelemetry collector, which can forward them to New Relic via OTLP.
</td>
<td>
[OpenTelemetry collector](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-best-practices-logs)
</td>
</tr>
</tbody>
</table>

For more on log forwarding options and specific use cases, see [Forward logs](/docs/logs/forward-logs/enable-log-management-new-relic).


<DocTiles numbered>
<DocTile title='Get started' label={{text: 'You are here', color: '#FCD672'}} path="/docs/journey-large-logs/get-started" ></DocTile>
<DocTile title='Filter and reduce your log ingest' path="/docs/journey-large-logs/filter" ></DocTile>
</DocTiles>

<DocTiles>
<DocTile title='Organize your logs' number='3'path="/docs/journey-large-logs/organize" ></DocTile>
<DocTile title='Use your logs' number='4' path="/docs/journey-large-logs/use" ></DocTile>
</DocTiles>

0 comments on commit a809f54

Please sign in to comment.