Skip to content

Commit

Permalink
fix(kubernetes): made structural changes
Browse files Browse the repository at this point in the history
  • Loading branch information
homelessbirds committed May 3, 2024
1 parent 4af5332 commit 246d433
Showing 1 changed file with 14 additions and 19 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -41,36 +41,31 @@ To forward your Kubernetes logs to New Relic with our plugin:

## Troubleshoot your Kubernetes plugin for log forwarding installation [#troubleshoot-installation]

Sometimes, despite correctly installing the Kubernetes plugin for log forwarding (`newrelic-logging` [Helm chart](https://github.com/newrelic/helm-charts/blob/master/charts/newrelic-logging)), you may encounter performance issues that affect the correct delivery of logs. In such circumstances, it can be helpful to look at the log forwarder internal metrics to understand what is the cause of a potential bottleneck.
Sometimes, despite correctly installing the Kubernetes plugin for log forwarding (`newrelic-logging` [Helm chart](https://github.com/newrelic/helm-charts/blob/master/charts/newrelic-logging)), you may encounter performance issues that affect the correct delivery of logs. In such circumstances, looking at the log forwarder internal metrics can be helpful to understand the cause of a potential bottleneck.

To this end, the `newrelic-logging` Helm chart provides a configuration setting to enable the collection of such metrics for a given Kubernetes cluster. Moreover, we provide a dashboard template in JSON format to easily display all these metrics in New Relic.
The `newrelic-logging` Helm chart provides a configuration setting to enable the collection of such metrics for a given Kubernetes cluster. Our JSON-formatted dashboard template easily displays all these metrics in New Relic.

To configure your Kubernetes cluster to send the log forwarder internal metrics and represent them in a dashboard, follow these steps:

1. Install the Helm chart with the following extra configuration setting:
```
newrelic-logging:

```
newrelic-logging:
fluentBit:
sendMetrics: true
```
<Callout variant="tip">
Only enable the `newrelic-logging.fluentBit.sendMetrics` when troubleshooting a Kubernetes cluster. We recommend enabling it for a single Kubernetes cluster at a time to ease troubleshooting.
</Callout>

```
You only need to enable the `newrelic-logging.fluentBit.sendMetrics` when troubleshooting a Kubernetes cluster. We recommend enabling it for a single Kubernetes cluster at a time to ease troubleshooting.
2. Download [this dashboard template file](https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/fluent-bit-and-plugin-metrics-dashboard-template.json). Open it in a text editor and replace all the `YOUR_ACCOUNT_ID` occurrences (49 in total) by your [New Relic Account ID](/docs/accounts/accounts-billing/account-structure/account-id/).

3. Import the resulting dashboard in JSON format by following [these instructions](/docs/query-your-data/explore-query-data/dashboards/dashboards-charts-import-export-data/#import-json).

4. The imported dashboard will be available in your [Dashboards page](https://one.newrelic.com/dashboards) as `Kubernetes Fluent Bit monitoring`.

<Callout variant="tip">
The `newrelic-logging` Helm chart uses [Fluent Bit](https://fluentbit.io/) together with New Relic's [newrelic-fluent-bit-output plugin]((https://github.com/newrelic/newrelic-fluent-bit-output)) to send logs to New Relic. The `fluentBit.sendMetrics` configuration option enables the collection of their individual metrics:
- **[Fluent Bit internal metrics](https://docs.fluentbit.io/manual/administration/monitoring#for-v2-metrics)**: emitted by Fluent Bit in Prometheus format and delivered to New Relic's Prometheus Export endpoint. **These are the recommended ones to troubleshoot a Kubernetes cluster** because they can be faceted by `cluster_name`, `node_name` and `hostname` (pod name).
- **[newrelic-fluent-bit-output's internal plugin metrics](https://github.com/newrelic/newrelic-fluent-bit-output?tab=readme-ov-file#troubleshooting-metrics)**: collected by the output plugin and sent to New Relic's Metric API. These metrics do not contain dimensions, so they cannot be narrowed down to a particular cluster or host. Despite this, it is useful to capture them for a single cluster to assess what is the overall latency when delivering the logs to the New Relic Logs API or to observe potential packaging problems.
</Callout>
### Configure Fluent Bit
The `newrelic-logging` Helm chart uses [Fluent Bit](https://fluentbit.io/) together with New Relic's [newrelic-fluent-bit-output plugin](https://github.com/newrelic/newrelic-fluent-bit-output) to send logs to New Relic. The `fluentBit.sendMetrics` configuration option enables the collection of their individual metrics:

* **[Fluent Bit internal metrics](https://docs.fluentbit.io/manual/administration/monitoring#for-v2-metrics)**: emitted by Fluent Bit in Prometheus format and delivered to New Relic's Prometheus Export endpoint. **These are the recommended ones to troubleshoot a Kubernetes cluster** because they can be faceted by `cluster_name`, `node_name` and `hostname` (pod name).
* **[newrelic-fluent-bit-output's internal plugin metrics](https://github.com/newrelic/newrelic-fluent-bit-output?tab=readme-ov-file#troubleshooting-metrics)**: collected by the output plugin and sent to New Relic's Metric API. These metrics do not contain dimensions, so they cannot be narrowed down to a particular cluster or host. Despite this, capturing them for a single cluster is useful to assess the overall latency when delivering the logs to the New Relic Logs API or to observe potential packaging problems.

<Callout variant="tip">
We capture Fluent Bit's internal metrics by using its [prometheus_scrape INPUT plugin](https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-scrape-metrics) in conjunction with its [prometheus_remote_write OUTPUT plugin](https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-remote-write). All the Prometheus `counter` metrics are actually _cumulative counters_, but we automatically perform a _delta conversion_ when they are ingested at New Relic to ease querying them using NRQL later. You can find more details [here](/docs/data-apis/understand-data/metric-data/cumulative-metrics/).
</Callout>
We capture Fluent Bit's internal metrics by using its [prometheus_scrape INPUT plugin](https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-scrape-metrics) in conjunction with its [prometheus_remote_write OUTPUT plugin](https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-remote-write). All the Prometheus `counter` metrics are actually _cumulative counters_, but we automatically perform a _delta conversion_ when they are ingested at New Relic to ease querying them using NRQL later. You can find more details [here](/docs/data-apis/understand-data/metric-data/cumulative-metrics/).

## View log data [#find-data]

Expand Down

0 comments on commit 246d433

Please sign in to comment.