Skip to content

Latest commit

 

History

History
1198 lines (916 loc) · 52.5 KB

forward-your-logs-using-infrastructure-agent.mdx

File metadata and controls

1198 lines (916 loc) · 52.5 KB
title tags translate metaDescription redirects freshnessValidatedDate
Forward your logs using the infrastructure agent
Logs
Enable log management in New Relic
Enable log monitoring in New Relic
kr
How to forward your logs to New Relic using our infrastructure agent, so you can use enhanced log management capabilities.
/docs/logs/enable-new-relic-logs/1-enable-logs/forward-your-logs-using-new-relic-infrastructure-agent
/docs/logs/enable-new-relic-logs/1-enable-logs/forward-your-logs-using-new-relic-infrastructure
/docs/logs/enable-log-monitoring-new-relic/enable-log-monitoring-new-relic/forward-your-logs-using-infrastructure-agent
/docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/forward-your-logs-using-infrastructure-agent
never

import logsHostLogsUi from 'images/logs_screenshot-full_host-logs-ui.webp'

import infrastructureAmazonLinux from 'images/infrastructure_logo_Amazon-linux.webp'

import infrastructureCentos from 'images/infrastructure_logo_centos.webp'

import infrastructureDocker from 'images/infrastructure_logo_docker.webp'

import infrastructureDebian from 'images/infrastructure_logo_debian.webp'

import infrastructureRedhat from 'images/infrastructure_logo_redhat.webp'

import infrastructureSuse from 'images/infrastructure_logo_suse.webp'

import infrastructureUbuntu from 'images/infrastructure_logo_ubuntu.webp'

import infrastructureWindows from 'images/infrastructure_logo_windows.webp'

import osAnsibleRed from 'images/os_icon_ansible-red.webp'

Forwarding your logs to New Relic makes all of your logging data available in one location, providing deeper visibility into both your application and your platform performance data. With your logs in one spot, you can collect, process, explore, query, and alert on errors or anomalies found in your log data.

Screenshot of logs in context for a host

From your host's UI, your logs are placed in context of events for the selected time period. You can drill down into detailed data for any of the highlighted attributes.

How you forward your logs depends on how you installed the infrastructure agent, since the infrastructure agent enables the log forwarding feature. You can install the infrastructure agent via:

  • Guided install (recommended for most users)
  • Manual installation
  • Linux tarball
The Linux version of infrastructure agent, specifically version 1.42.0, has transitioned from using the td-agent-bit package to the fluent-bit package. This change was necessitated by the fact that fluent-bit is no longer distributed in the td-agent-bit flavour after the major version 2.x update.

To ensure smooth operations and provide the option to revert to td-agent-bit in the event of any issues with the fluent-bit package, the infrastructure agent now installs both packages (td-agent-bit and fluent-bit). By default, the agent is configured to use fluent-bit.

See Rollback to Fluent Bit 1.9 for more information about how to rollback.

Got lots of logs? Check out our [tutorial on how to optimize and manage them](/docs/tutorial-large-logs/get-started-managing-large-logs/).

System requirements [#system]

  • Infrastructure agent version 1.11.4 or higher
  • Fluent Bit. The infrastructure agent already installs the latest version for you. To update or downgrade it to a specfic version, refer to the Fluent Bit installation procedures.
  • OpenSSL library 1.1.0 or higher
  • Built-in support for ARM64 architecture on Linux systems (for example, AWS Graviton architecture) added in infrastructure agent 1.20.6.
  • Amazon Linux 2 and 2023 (ARM64 not supported for 2023)
  • CentOS version 7, 8 and 9 Stream (Rocky Linux and AlmaLinux are also supported)
  • RedHat version 7, 8 and 9
  • Debian version 9 (Strech), 10 (Buster) and 11 (Bullseye).
  • SUSE Linux Enterprise Server (SLES) version 12 and 15 (ARM64 not supported).
  • Ubuntu versions 16.04.x, 18.04.x, 20.04.x, 22.04.x (LTS versions).
    • Ubuntu 24.04.x LTS is not supported yet.
  • Windows Server 2012, 2016, 2019, and 2022, and their service packs.
  • Windows 10, Windows 11.

Automatically forward logs with guided install [#infra]

When you use our guided install to install the infrastructure agent, the log forwarding feature is automatically configured during the install process. If you haven't already, create your free New Relic account below to start monitoring your data today.

To initiate your install, choose your deployment method:

} />

<TechTile name="Ansible" to="/docs/infrastructure/new-relic-infrastructure/config-management-tools/configure-new-relic-infrastructure-using-ansible" icon={Ansible} />

<TechTile name="CentOS" to="https://one.newrelic.com/marketplace?state=8f14e646-461e-b010-4675-3a0658bb3d20" icon={CentOS} />

<TechTile name="Debian" to="https://one.newrelic.com/marketplace?state=8f14e646-461e-b010-4675-3a0658bb3d20" icon={Debian} />

<TechTile name="RHEL" to="https://one.newrelic.com/marketplace?state=8f14e646-461e-b010-4675-3a0658bb3d20" icon={Red Hat} />

<TechTile name="SLES" to="https://one.newrelic.com/marketplace?state=8f14e646-461e-b010-4675-3a0658bb3d20" icon={SLES} />

<TechTile name="Ubuntu" to="https://one.newrelic.com/marketplace?state=8f14e646-461e-b010-4675-3a0658bb3d20" icon={Ubuntu} />

<TechTile name="Windows" to="https://one.newrelic.com/marketplace?state=a792b092-300f-a5a4-ff62-510f4e0c52a5" icon={Windows} />

Enable log forwarding on agent installed manually [#manual]

To install the infrastructure agent manually, follow our tutorial to install the package manager, or check out our MSI installer (Windows).

Step 1. Configure the infrastructure agent

Configuration files direct your system to forward the log sources you want to appear in New Relic. You can add as many config files as you want. Our infrastructure agent uses .yml files to configure logging. If you install the infrastructure agent via Add data in the UI, the file logging.yml is created automatically.

To add a new configuration file for the log forwarding feature:

  1. Navigate to the log forwarder configuration folder:

    • Linux: /etc/newrelic-infra/logging.d/
    • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\
  2. Create a logging.yml configuration file, and add the parameters you need. The logging.d directory has various .yml.example files you can use as a reference or starting point. For Windows examples, see our Github repo.

# Log forwarder configuration file example
# Source: file
# Available customization parameters: attributes, max_line_kb, pattern
logs:
  # Basic tailing of a single file
  - name: basic-file
    file: /var/log/logFile.log

  # File with spaces in its path. No need to use quotes.
  - name: file-with-spaces-in-path
    file: /var/log/folder with spaces/logFile.log

  # Specify a list of custom attributes, as key-value pairs, to be included
  # in each log record
  - name: file-with-attributes
    file: /var/log/logFile.log
    attributes:
      application: tomcat
      department: sales
      maintainer: example@mailprovider.com

  # Use wildcards to refer to multiple files having a common extension or
  # prefix. Newly generated files will be automatically detected every 60
  # seconds.
  #
  # WARNING: avoid using wildcards that include the file extension, since
  # it'll cause logs to be forwarded repeatedly if log rotation is enabled.
  - name: log-files-in-folder
    file: /var/log/logF*.log

  # Lines longer than 128 KB will be automatically skipped. Use 'max_line_kb'
  # to increase this limit.
  - name: log-file-with-long-lines
    file: /var/log/logFile.log
    max_line_kb: 256

  # Use 'pattern' to filter records using a regular expression
  - name: only-records-with-warn-and-error
    file: /var/log/logFile.log
    pattern: WARN|ERROR

The agent automatically processes new configuration files without having to restart the infrastructure monitoring service. The only exception to this is when configuring a custom Fluent Bit configuration.

Step 2. Set log forwarding parameters [#parameters]

You're required to set up a name and log source parameter in the log forwarding .yml config file. To start, define a name of the log or logs you want to forward to New Relic.

What you use for the log source depends on the location your logs are sourced from. Available options for the log source include:

Path to the log file or files. The agent tracks changes on the log files in a way similar to `tail -f` shell.
<DNT>**Example:**</DNT>

```yml
  logs:
  - name: example-log
    file: /var/log/example.log     # Path to a single log file
  - name: example-log-two
    file: /var/log/example-two.log # Path to another single log file
```

The `file` parameter can point to a specific log file or multiple files by using wildcards applied to names and extensions; for example, `/logs/*.log`. You can use wildcards in place of directories in a file path, which can be used to tail files located in different directories.

<DNT>**Example:**</DNT>

```yml
logs:
  - name: docker-logs
    file: /var/lib/docker/containers/*/*.log # Path to multiple folders and files
```

<Callout variant="important">
  Use of wildcards may significantly increase the number of file descriptors and inotify watches the Fluent Bit process keeps open, which can interfere with log collection if the host's file descriptor limit is reached. Tailing a large number of files may require you to increase the maximum number of file descriptors and inotify watchers allowed by the operating system. Please refer to [Errors when tailing a large amount of log files](#too-many-files) for more details on how to increase them.
</Callout>

<Collapser id="systemd" title="systemd"

Use the `systemd` parameter to forward log messages that are collected by the `journald` daemon in Linux environments. This input type requires the agent to run in [root mode](/docs/infrastructure/install-configure-infrastructure/linux-installation/linux-agent-running-modes).

<DNT>**Example:**</DNT>

```yml
logs:
  - name: systemd-example
    systemd: cupsd
```

<Collapser id="syslog" title="syslog"

Syslog data source.

<DNT>**Parameters:**</DNT>

* `uri:` Syslog socket. Format varies depending on the protocol:
  * TCP/UDP network sockets: `[tcp/udp]://LISTEN_ADDRESS:PORT`
  * Unix domain sockets: `unix_[tcp/udp]:// + /socket/path`
* `parser:` Syslog parser. Default is `rfc3164`. Use `rfc5424` if your messages include fractional seconds. Note: `rfc3164` currently does not work on SuSE.
* `unix_permissions:` Default is `0644` for domain sockets. This limits entries to processes running as root. You can use `0666` to listen for non-root processes (at your own risk).

  When running the agent in [privileged mode](/docs/infrastructure/install-configure-infrastructure/linux-installation/linux-agent-running-modes), ports and sockets must be available or owned by `nri-agent` and with `0666` file permissions so that other processes can write logs to the sockets.

  ```yml
  logs:
    # TCP network socket
    - name: syslog-tcp-test
      syslog:
        uri: tcp://0.0.0.0:5140 # Use the tcp://LISTEN_ADDRESS:PORT format
        parser: rfc5424 # Default syslog parser is rfc3164

    # UDP network socket
    - name: syslog-udp-test
      syslog:
        uri: udp://0.0.0.0:6140 # Use the udp://LISTEN_ADDRESS:PORT format
      max_line_kb: 35

    # Unix TCP domain socket
    - name: syslog-unix-tcp-test
      syslog:
        uri: unix_tcp:///var/unix-tcp-socket-test
        unix_permissions: 0666 # Default is 0644. Change at your own risk

    # Unix UDP domain socket
    - name: syslog-unix-udp-test
      syslog:
        uri: unix_udp:///var/unix-udp-socket-test
        parser: rfc5424
  ```

<Collapser id="tcp" title="tcp"

Logs retrieved over TCP connections.

<DNT>**Parameters:**</DNT>

* `uri:` TCP/IP socket to listen for incoming data. The URI format is `tcp://LISTEN_ADDRESS:PORT`.
* `format:` Format of the data. It can be `json` or `none`.
* `separator:` If `format: none` is used, you can define a separator string for splitting records (default: `\n`).

  ```yml
  logs:
    - name: tcp-simple-test
      tcp:
        uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format
        format: none            # Raw text - this is default for 'tcp'
        separator: \t           # String for separating raw text entries
      max_line_kb: 32
    - name: tcp-json-test
      tcp:
        uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format
        format: json
  ```

<Collapser id="winevtlog" title="winevtlog"

<Callout variant="important">
  Available since infrastructure agent v.1.24.3
  Only compatible with Windows Server 2019 and later. Use [winlog](/docs/logs/forward-logs/forward-your-logs-using-infrastructure-agent/#winlog) instead for prior versions.
</Callout>

Collect event from Windows log channels using new Windows Event Log API using the [winevtlog Fluent Bit plugin](https://docs.fluentbit.io/manual/pipeline/inputs/windows-event-log-winevtlog).

<DNT>**Parameters:**</DNT>

* `channel`: Name of the channel logs will be collected from.
* `collect-eventids`: List of Windows Event IDs to be collected and forwarded to New Relic. Event ID ranges are supported.
* `exclude-eventids`: List of Windows Event IDs to be excluded from collection. Event ID ranges are supported.
* `use-ansi`: Use ANSI encoding on winlog messages. We use ANSI encoding by default on Windows Server 2016 and older versions, and UTF-8 on newer ones. You can override this behavior with this configuration parameter if these defaults do not suit your use case. This solves the problem of ANSI character code logs being empty strings. It's not a function to convert multi-byte character code logs in ANSI character code to UTF-8.

  All events are collected from the specified channel by default. Configure the `collect-eventids` and `exclude-eventids` sections to avoid sending unwanted logs to your New Relic account.

  Add event IDs or ranges to `collect-eventids` or `exclude-eventids` to forward or drop specific events. `exclude-eventids` takes precedence over `collect-eventids` if the same event ID is present in both sections.

  <DNT>**Example:**</DNT>

  ```yml
  logs:
    # Example winevtlog security log ingestion with eventId filters.
    - name: windows-security
      winevtlog:
        channel: Security
        collect-eventids:
          - 4624
          - 4265
          - 4700-4800
        exclude-eventids:
          - 4735
      attributes:
        logtype: windows_security

    # Example entries for the application and system channels
    - name: windows-application
      winevtlog:
        channel: Application
        attributes:
          logtype: windows_application

    # Example entries for the application use-ansi
    - name: windows-application
      winevtlog:
        channel: Application
        attributes:
          logtype: windows_application
        use-ansi: true

    - name: windows-system
      winevtlog:
        channel: System
        attributes:
          logtype: windows_system

    # Example/Optional entry for Windows Defender Logs
    - name: windows-defender
      winevtlog:
        channel: Microsoft-Windows-Windows Defender/Operational

    # Example/Optional entry for Windows Clustering Logs
    - name: windows-clustering
      winevtlog:
        channel: Microsoft-Windows-FailoverClustering/Operational
  ```

<Collapser id="winlog" title="winlog"

<Callout variant="important">
Winlog can only collect classic Event logs. Attempting to capture others will silently collect the Applications log.
</Callout>
Collect events from Windows log channels.

<DNT>**Parameters:**</DNT>

* `channel`: Name of the channel logs will be collected from. It doesn't work for custom channels.
* `collect-eventids`: List of Windows Event IDs to be collected and forwarded to New Relic. Event ID ranges are supported.
* `exclude-eventids`: List of Windows Event IDs to be excluded from collection. Event ID ranges are supported.
* `use-ansi`: Use ANSI encoding on winlog messages. We use ANSI encoding by default on Windows Server 2016 and older versions, and UTF-8 on newer ones. You can override this behavior with this configuration parameter if these defaults do not suit your use case. This solves the problem of ANSI character code logs being empty strings. It's not a function to convert multi-byte character code logs in ANSI character code to UTF-8.

  All events are collected from the specified channel by default. Configure the `collect-eventids` and `exclude-eventids` sections to avoid sending unwanted logs to your New Relic account.

  Add event IDs or ranges to `collect-eventids` or `exclude-eventids` to forward or drop specific events. `exclude-eventids` takes precedence over `collect-eventids` if the same event ID is present in both sections.

  <DNT>**Example:**</DNT>

  ```yml
  logs:
    # Example winlog security log ingestion with eventId filters.
    - name: windows-security
      winlog:
        channel: Security
        collect-eventids:
          - 4624
          - 4265
          - 4700-4800
        exclude-eventids:
          - 4735
      attributes:
        logtype: windows_security

    # Example entries for the application and system channels
    - name: windows-application
      winlog:
        channel: Application
        attributes:
          logtype: windows_application
    - name: windows-system
      winlog:
        channel: System
        attributes:
          logtype: windows_system

    # Example entries for the application use-ansi
    - name: windows-application
      winlog:
        channel: Application
        attributes:
          logtype: windows_application
        use-ansi: true

    # Example/Optional entry for Windows Defender Logs
    - name: windows-defender
      winlog:
        channel: Microsoft-Windows-Windows Defender/Operational

    # Example/Optional entry for Windows Clustering Logs
    - name: windows-clustering
      winlog:
        channel: Microsoft-Windows-FailoverClustering/Operational
  ```

Step 3. Define key attributes [#optional-config]

Although these configuration parameters aren't required, we still recommend you apply these configurations to your logging.yml file so you get the most out of log forwarding.

List of custom attributes specified as key-value pairs that can be used to send additional data with the logs which you can then query. The `attributes` configuration parameter can be used with any log source.
<Callout variant="important">
  The `attributes` configuration parameter does not add custom attributes to logs forwarded via external Fluent Bit configuration (for example, using the `fluentbit` configuration parameter). In this scenario, you should refer to the `record_modifier` option in the [Fluent Bit documentation](https://docs.fluentbit.io/manual/).
</Callout>

One common use of the `attributes` configuration parameter is to specify the `logtype` attribute. This attribute allows leveraging one of the [built-in parsing rules](/docs/logs/log-management/ui-data/parsing/#built-in-rules) supported by New Relic's <InlinePopover type="logs" /> capabilities.

<DNT>**Example:**</DNT>

```yml
logs:
  - name: example-file-attributes
    file: /var/log/example.log
    attributes:
      logtype: nginx
      region: example-us-02
      team: A-team
  - name: example-tcp-attributes
    tcp:
      uri: tcp://0.0.0.0:2345
      format: json
    attributes:
      logtype: nginx
      region: example-us-02
      team: B-team
```

<Collapser id="automatically-inserted-attributess" title="attributes automatically inserted by the infrastructure agent"

The infrastructure agent automatically inserts log attributes for your convenience. Some of them are inserted for any log record, while others depend on the configuration parameters you used while setting up the log forwarder.

<table>
  <thead>
    <tr>
      <th style={{ width: "200px" }}>
        Attribute name
      </th>

      <th>
        Description
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        `entity.guids`
      </td>

      <td>
        Always inserted.

        The infrastructure agent inserts the [Entity GUID](/attribute-dictionary/?event=Span&attribute=entityGuid) assigned by New Relic to identify the host where it's running. It is available in the `entity.guids` field.

        Note: If the captured logs belong to an application instrumented using APM, the `entity.guids` field contains both the entity GUID of infrastructure, as well as the GUID of APM, separated by a pipe ( | ) delimiter.
      </td>
    </tr>

    <tr>
      <td>
        `fb.input`
      </td>

      <td>
        Always inserted.

        The underlying [Fluent Bit input plugin type](https://docs.fluentbit.io/manual/pipeline/inputs) used to capture the logs. Currently its values are `tail`, `systemd`, `winlog`, `syslog`, and `tcp`.
      </td>
    </tr>

    <tr>
      <td>
        `filePath`
      </td>

      <td>
        Inserted when using the `file` input type.

        Absolute file path of the file being monitored.
      </td>
    </tr>

    <tr>
      <td>
        `hostname`
      </td>

      <td>
        Always inserted.

        The hostname of the machine/VM/container executing the infrastructure agent.
      </td>
    </tr>

    <tr>
      <td>
        `plugin.type`
      </td>

      <td>
        Always inserted.

        Indicates the utility used to capture the logs. In this case, it is the infrastructure agent itself, so this attribute always has the value `nri-agent`.
      </td>
    </tr>
  </tbody>
</table>

<Collapser id="pattern" title="pattern"

Regular expression for filtering records. Only supported for the `tail`, `systemd`, `syslog`, and `tcp` (only with format `none`) sources.

This field works in a way similar to `grep -E` in Unix systems. For example, for a given file being captured, you can filter for records containing either `WARN` or `ERROR` using:

```yml
- name: only-records-with-warn-and-error
  file: /var/log/logFile.log
  pattern: WARN|ERROR
```

No filtering is applied by default.

<Collapser id="max_line_kb" title="max_line_kb"

Maximum size of log entries/lines in KB. If log entries exceed the limit, they are skipped. Default is `128`,the minimum allowed value is `33`.

<Collapser id="fluentbit" title="fluentbit"

External [Fluent Bit](https://fluentbit.io/) configuration and parser files. If defined, they are merged with the existing configuration and parser files generated by the infrastructure agent.

The infrastructure agent processes the configuration files located in the `logging.d` directory and will generate a run-time Fluent Bit configuration file that contains the appropriate `[INPUT]`, `[FILTER]` and `[OUTPUT]` sections. Optionally, it will also declare an `@INCLUDE` in case you provided an external Fluent Bit configuration file via the `fluentbit` option.

The runtime file does not define a [`[SERVICE]` section](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file), leaving all default Fluent Bit configuration values. You can still override Fluent Bit's default settings by defining your own `[SERVICE]` section in your external Fluent Bit configuration file and include it via the `fluentbit` option.

<DNT>**Parameters:**</DNT>

`config_file:` path to an existing Fluent Bit configuration file. Note that any overlapping source results in duplicate messages in our logs UI.

`parsers_file:` path to an existing Fluent Bit parsers file. The following parser names are reserved: `rfc3164`, `rfc3164-local` and `rfc5424`.

<Callout variant="important">
  The infrastructure agent allows forwarding logs for the most common use cases by defining simple log forwarding configurations in the YAML files in the `logging.d/` directory, as described in this document. These files are internally translated into Fluent Bit configuration files with the correct format and sane configuration defaults. New Relic provides official support for these configuration options since we ensure that the generated configuration files are correct and operative.

  Nevertheless, for those use cases not covered by our supported configuration options, we provide the possibility to use an externally-generated Fluent Bit configuration and parsers file using the `fluentbit`, `config_file`, and `parsers_file` options.

  Note: We cannot guarantee the correct operation of the log forwarded in this case, given that the provided configurations are completely arbitrary and are not being generated/validated by the agent. Therefore, New Relic does not provide official support for the external configurations specified via these options.
</Callout>

Sample configuration file [#running-modes]

Here is an example of a logging.d configuration file in YAML format. For more configuration examples, see the infrastructure agent repository.

```yml # Remember to only use spaces for indentation
logs:
  # Example of 'file' source
  - name: file-with-attributes
    file: /var/log/test.log # Path to a single file or pattern
    attributes:             # You can use custom attributes to enrich your data
      logtype: nginx
      team: The A Team
    pattern: Error          # Regular expression to filter log entries

  # Example of 'systemd' source (Linux only)
  - name: systemd-example
    systemd: cupsd

  # Examples of 'syslog' source, one per protocol
  # TCP network socket
  - name: syslog-tcp-test
    syslog:
      uri: tcp://0.0.0.0:5140 # Use the tcp://LISTEN_ADDRESS:PORT format
      parser: rfc5424         # Default syslog parser is rfc3164

  # UDP network socket
  - name: syslog-udp-test
    syslog:
      uri: udp://0.0.0.0:6140 # Use the udp://LISTEN_ADDRESS:PORT format
    max_line_kb: 35

  # Paths for Unix sockets are defined by combining protocol and path:
  # unix_udp:// + /path/socket - for example, unix_udp:///tmp/socket
  # Unix TCP domain socket
  - name: syslog-unix-tcp-test
    syslog:
      uri: unix_tcp:///var/unix-tcp-socket-test
      unix_permissions: 0666 # Default is 0644. Change at your own risk

  # Unix UDP domain socket
  - name: syslog-unix-udp-test
    syslog:
      uri: unix_udp:///var/unix-udp-socket-test
      parser: rfc5424

  # Examples of 'tcp' source for formats 'none' and 'json'
  - name: tcp-simple-test
    tcp:
      uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: none            # Raw text - this is default for 'tcp'
      separator: \t           # String for separating raw text entries
    attributes:               # You can add custom attributes to any source of logs
      tcpFormat: none
      someOtherAttribute: associatedValue
    max_line_kb: 32
  - name: tcp-json-test
    tcp:
      uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format
      format: json
    attributes:
      tcpFormat: json
      yetAnotherAttribute: 12345

  # Example of Fluent Bit configuration import
  - name: fluentbit-import
    fluentbit:
      config_file: /path/to/fluentbit.config
      parsers_file: /path/to/fluentbit/parsers.conf
```

Step 4. View your log data [#find-data]

If everything is configured correctly and your data is being collected, you should see logs and related telemetry data in these places:

  • Our logs UI
  • In the infrastructure UI, from the host table, click the icon for a specific host, and then click View logs.
  • Our tools for running NRQL queries. For example, you can execute a query like this:
SELECT * FROM Log

Enable logging for your on-host integrations [#on-host]

With the infrastructure agent installed, you can enable automatic log parsing and forwarding for our most popular on-host integrations with one step. To enable this feature, rename the on-host-log.yml.example file to on-host-log.yml. Once done, your integration's logs are automatically parsed and sent to New Relic.

This option is available for our supported Linux platforms.

To enable the on-host integration log forwarding feature:

Copy or rename the `elasticsearch-log.yml.example` file to `elasticsearch-log.yml` to enable automatic Elasticsearch JSON formatted log parsing and forwarding to New Relic. No need to restart the agent.
<DNT>**Example:**</DNT>

```bash
sudo cp /etc/newrelic-infra/logging.d/elasticsearch-log.yml.example /etc/newrelic-infra/logging.d/elasticsearch-log.yml
```

<Collapser id="mysql-logs" title="MySQL logs"

Copy or rename the `mysql-log.yml.example` file to `mysql-log.yml` to enable automatic MySQL error log parsing and forwarding to New Relic. No need to restart the agent.

<DNT>**Example:**</DNT>

```bash
sudo cp /etc/newrelic-infra/logging.d/mysql-log.yml.example /etc/newrelic-infra/logging.d/mysql-log.yml
```

<Collapser id="nginx-logs" title="NGINX logs"

Copy or rename the `nginx-log.yml.example` file to `nginx-log.yml` to enable automatic NGINX access and error log parsing and forwarding to New Relic. No need to restart the agent.

<DNT>**Example:**</DNT>

```bash
sudo cp /etc/newrelic-infra/logging.d/nginx-log.yml.example /etc/newrelic-infra/logging.d/nginx-log.yml
```

<Collapser id="rabbitmq-logs" title="Rabbitmq logs"

Copy or rename the `rabbitmq-log.yml.example` file to `rabbitmq-log.yml` to enable automatic Rabbitmq error log parsing and forwarding to New Relic. No need to restart the agent.

<DNT>**Example:**</DNT>

```bash
sudo cp /etc/newrelic-infra/logging.d/rabbitmq-log.yml.example /etc/newrelic-infra/logging.d/rabbitmq-log.yml
```

<Collapser id="redis-logs" title="Redis logs"

Copy or rename the `redis-log.yml.example` file to `redis-log.yml` to enable automatic Redis error log parsing and forwarding to New Relic. No need to restart the agent.

<DNT>**Example:**</DNT>

```bash
sudo cp /etc/newrelic-infra/logging.d/redis-log.yml.example /etc/newrelic-infra/logging.d/redis-log.yml
```

Enable log forwarding on agent installed using Linux tarball [#tarball-install]

Our custom Linux installation process for infrastructure monitoring allows you to tailor all aspects of the installation process, and to place files and folders on your machine. If you choose the assisted or manual tarball installation process, follow these steps to implement the log forwarder feature:

  1. Create the following directories:

    • /var/db/newrelic-infra/newrelic-integrations/logging
    • /etc/newrelic-infra/logging.d
  2. Download and install New Relic's fluent-bit-package (RPM) by running a command similar to:

    yum localinstall fluent-bit-<some-version>.rpm
  3. Download New Relic's fluentbit plugin and save it as /var/db/newrelic-infra/newrelic-integrations/logging/out_newrelic.so.

  4. Download or copy the parsers.conf file from this Github repository, and save it as /var/db/newrelic-infra/newrelic-integrations/logging/parsers.conf.

Troubleshooting [#troubleshoot]

If you encounter problems with configuring your log forwarder, try these troubleshooting tips.

If no data appears after you enable our log management capabilities, follow our [standard log troubleshooting procedures](/docs/logs/log-management/troubleshooting/no-log-data-appears-ui/).

<Collapser className="freq-link" id="no-data" title="No data appears when tailing a file"

The log forwarding feature requires the agent to have permission to read the data sources. When running the infrastructure agent in [privileged or non-privileged modes](/docs/infrastructure/install-configure-infrastructure/linux-installation/linux-agent-running-modes), make sure that the log files you want to forward (and any intermediary directory in its path) are readable by the user running `nri-agent`.

<DNT>**Example: Check file access under Linux**</DNT>

Let's check whether the file `/var/log/restrictedLogs/logFile.log` can be monitored by the `nri-agent` user. In Linux, you can do a quick check with the `namei` command:

```bash
sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
[output] f: /var/log/restrictedLogs/logFile.log
[output] drwxr-xr-x root root /
[output] drwxr-xr-x root root var
[output] drwxrwxr-x root syslog log
[output] drwxr--r-- root root restrictedLogs
[output] logFile.log - No such file or directory
```

This command failed because the file is not visible to the `nri-agent` user. By inspecting the previous output, we can detect that the `restrictedLogs` directory is missing the execution flag for `others`.

To fix this, execute:

```bash
sudo chmod 755 /var/log/restrictedLogs
```

And then check for file access again:

```bash
sudo -u nri-agent namei -ml /var/log/restrictedLogs/logFile.log
[output] f: /var/log/restrictedLogs/logFile.log
[output] drwxr-xr-x root root /
[output] drwxr-xr-x root root var
[output] drwxrwxr-x root syslog log
[output] drwxr-xr-x root root restrictedLogs
[output] -rw-r----- vagrant vagrant logFile.log
```

The file is now visible to the `nri-agent` user. You must ensure that the file is also readable by the `nri-agent` user. To check this, use:

```bash
sudo -u nri-agent head /var/log/restrictedLogs/logFile.log
[output] head: cannot open '/var/log/restrictedLogs/logFile.log' for reading: Permission denied
```

In this example, the file is missing the read rights for the `others` group (users other than `vagrant` and the `vagrant` user group). You could fix this by granting read permissions to `others`, but the application could change these permissions upon restart.

To avoid this, a better approach is to add the `nri-agent` user to the `vagrant` user group.

<Collapser className="freq-link" id="syslog" title="No data appears when capturing via a Syslog socket"

The log forwarding feature requires that the agent has permission to read the data sources. When running the infrastructure agent in [privileged or non-privileged modes](/docs/infrastructure/install-configure-infrastructure/linux-installation/linux-agent-running-modes):

* If you're using Unix domain socket files, make sure that the `nri-agent` user can access these files (please refer to the previous section) and that they have read and write permissions (`666`) so that other users than `nri-agent` can write to them.
* If you're using IP sockets, ensure that the port that you are using is not a system reserved one (like port `80`, for example).

  If no data appears after you enable log management, follow [standard log management troubleshooting procedures](/docs/logs/new-relic-logs/troubleshooting/no-data-appears-logs).

<Collapser className="freq-link" id="proxy" title="No data appears using infrastructure agent proxy"

As explained in the [infrastructure agent configuration guidelines](/docs/infrastructure/install-infrastructure-agent/configuration/infrastructure-agent-configuration-settings#proxy), the `proxy` parameter must use either HTTP or HTTPS and be in the form `https://user:password@hostname:port`. The agent can parse the parameter without the HTTP or HTTPS, but the log forwarder cannot. You will see an error like the following in the agent verbose logs:

```
[ERROR] building HTTP transport: parse \"hostname:port\":
first path segment in URL cannot contain colon
```

To solve this problem, check your `newrelic-infra.yml` file, and ensure the `proxy` parameter adheres to this form.

If you're using `caBundleFile` or `caBundleDir` in order to specify any certificate, we recommend to follow the below rules for each OS:

<DNT>**Linux**</DNT>
For `HTTP` proxies you don't need to setup any certificates. The plugin loads the system certificates and New Relic sends logs into the logging endpoint. However, you can specify the proxy self-signed certificate (PEM file) using either the `caBundleFile` or `caBundleDir` parameters.

<DNT>**Windows**</DNT>

* For `HTTP` proxies you don't need to setup any certificates. The plugin loads the system certificates.

* For `HTTPS`, you can configure it in one of the following ways:

    * (Recommended) Import the proxy certificate to the system pool. Import the proxy self-signed certificate (PEM file) by using the MMC tool. Refer to [this link](https://www.ssls.com/knowledgebase/how-to-import-intermediate-and-root-certificates-via-mmc/), and in <DNT>**Step 2**</DNT> ensure to import it in your `Trusted Root Certification Authorities` instead of in the `Intermediate Certification Authorities`.

    * Use the `caBundleFile` and `caBundleDir` parameters
  On Windows, we cannot load both the certificates from the system certificate pool and the ones specified with the `caBundleFile` `caBundleDir` parameters. So, if you are using `caBundleFile` or `caBundleDir`, ensure that the following certificates are placed in the same PEM file (when using `caBundleFile`) or in the same directory (when using `caBundleDir`):

* The proxy certificate (because it's an `HTTPS` proxy).

* The logging endpoint certificate (eg. `https://log-api.newrelic.com/log/v1`).

* The infrastructure agent certificate (eg. `https://infra-api.newrelic.com`).

  You can check the certificates by running:

  ```bash
  openssl s_client -connect log-api.newrelic.com:443 -servername log-api.newrelic.com
  ```

<Collapser className="freq-link" id="agent-logs" title="Sending the infrastructure agent's logs to New Relic"

You can configure the infrastructure agent to send its own logs to New Relic. This is useful for troubleshooting issues with log forwarding, the agent, or when contacting [support](https://support.newrelic.com/).
Trace logging generates a lot of data very quickly. To reduce disk space consumption and data ingest, when finished generating logs, be sure to set `level: info` (or lower).
To forward the infrastructure agent logs to New Relic:

1. Edit your `newrelic-infra.yml` file.
2. Enable logs forwarding to New Relic by adding the following config snippet:
   ```yml
   log:
     level: trace  # Recommended: Helps with troubleshooting
     forward: true # Enables sending logs to New Relic
     format: json  # Recommended: Enable agent logging in JSON format
     stdout: false # On Windows and systems that don't use `systemd` or where `journald` is inaccessible
   ```
3. [Restart the agent](/docs/infrastructure/new-relic-infrastructure/configuration/start-stop-restart-check-infrastructure-agent-status) to load the new settings.

This configuration sets up the agent in troubleshooting mode, but the log forwarder (based on Fluent Bit) will continue in a non-verbose mode.

<Collapser className="freq-link" id="fluentbit-logs" title="Enabling verbose mode in the log forwarder (Fluent Bit)"

Sometimes you can have issues with the log forwarder itself. For example, there may be problems accessing a specific channel when shipping Windows log events or when accessing a particular log file. In these situations, you can also enable the verbose mode for the log forwarder.
Trace logging generates a lot of data very quickly. To reduce disk space consumption and data ingest, when finished generating logs, be sure to set `level: info` (or lower).
1. Edit your `newrelic-infra.yml` file.
2. Enable Fluent Bit verbose logs by adding the following config snippet:
   ```yml
   log:
     level: trace
     forward: true # Enables sending logs to New Relic
     format: json  # Recommended: Enable agent logging in JSON format
     stdout: false # On Windows and systems that don't use `systemd` or where `journald` is inaccessible
     include_filters:
       traces:
         - supervisor # Required to see verbose logs from Fluent Bit
   ```
3. [Restart the agent](/docs/infrastructure/new-relic-infrastructure/configuration/start-stop-restart-check-infrastructure-agent-status) to load the new settings.

<Collapser className="freq-link" id="no-fb" title="Fluent Bit does not start with the infra agent"

<Callout variant="important">
  Fluent Bit's tail plugin does not support network drives.
</Callout>

For Linux versions prior to 2016, you may need to update the OpenSSL library to 1.1.0 (or higher). To check if you have this problem:

1. See if `infra-agent` has started Fluent Bit by running:

   ```bash
   ps -aux | grep fluent-bit
   ```

2. If it isn't running go to `/var/db/newrelic-infra/newrelic-integrations/logging` and run:

   ```bash
   ./fluent-bit -i systemd -o stdout
   ```

3. If you get the following error:

   ```
   error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
   ```

  Make sure to update OpenSSL to 1.1.0 or higher.

<Collapser className="freq-link" id="windows-runtime-error" title="Runtime error on Windows"

One of the following error messages may appear when enabling log forwarding on Windows:

```
The code execution cannot proceed because VCRUNTIME140.dll was not found.
```

OR

```
error="exit status 3221225781" process=log-forwarder
```

This is caused by a missing DLL.

To solve the issue, install the [Microsoft Visual C++ Redistributable](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) as applicable:

* [x64](https://aka.ms/vs/16/release/vc_redist.x64.exe)
* [x86](https://aka.ms/vs/16/release/vc_redist.x86.exe)

<Collapser className="freq-link" id="too-many-files" title="Errors when tailing a large amount of log files (Linux)"

It's common to face either one of the following error messages when attempting to tail a large amount of files:

  • Too many open files
  • The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource

The operating system defines a maximum amount of allocatable file descriptors (typically 1024 by default), and a maximum amount of allocatable inotify watches (typically 8192 by default). Any process attempting to go above these limits will fail, returning one of the errors above.

The underlying technology we use to forward logs, Fluent Bit, opens one file descriptor and sets an inotify watch for each file you configure to be forwarded. Moreover, at the time of writing this section, Fluent Bit uses an extra set of 32 file descriptors for its normal operation, with another extra file descriptor when it shuts down. Therefore, to capture a large amount of files you need to ensure that both the file descriptor and inotify watch limits are slightly greater than the amount of log files you wish to tail.

The following instructions summarize how to increase these limits if you want to tail 10,000 log files. Also, it assumes the infrastructure agent is installed in root running mode, and therefore must be run using the root user.

  1. Check which is the current hard limit for the amount of file descriptors per process. Typically, this limit should be quite high and should not need to be modified.

    ulimit -Hn
  2. Add the following line to /etc/security/limits.conf. We specified a limit of 10100 here instead of just 10000 to allow Fluent Bit to allocate the extra file descriptors it may need to work.

    root soft nofile 10100 # replace root by nri-agent for non-root (privileged and unprivileged) installations
  3. Add the following line to /etc/pam.d/common-session so that the previous limit is applied upon restart:

    session required pam_limits.so
  4. Add the following line to /etc/sysctl.conf to increase the amount of allowed inotify watchers per user. We specified a limit of 18192 here instead of just 10000 so that the root user will still have 8192 available inotify watches (the default value).

    fs.inotify.max_user_watches=18192
  5. Restart your system.

  6. Ensure that the new limits have been enforced by running:

    ulimit -Sn                                 # Should return 10100
    cat /proc/sys/fs/inotify/max_user_watches  # Should return 18192

    Learn more on how to increase open file limits, or on how to increase the inotify watches.

<Collapser className="freq-link" id="install-fb-version" title="Install the latest Fluent Bit version (Linux)"

Prior to version [1.19.0](/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes/new-relic-infrastructure-agent-1190) (or version [1.20.3](/docs/release-notes/infrastructure-release-notes/infrastructure-agent-release-notes/new-relic-infrastructure-agent-1203) for SLES 12.5), the Linux infrastructure agent came bundled with a Fluent Bit binary. Starting in this version, Fluent Bit is now included as as a separate `recommended` package dependency.

This means that you can install, update, or downgrade Fluent Bit separately from the agent. For your convenience, we have included several Fluent Bit packages in the same repository where the infrastructure lives, so you don't need to install any additional repositories to upgrade Fluent Bit.

Note that the agent automatically installs Fluent Bit when you first install it, using the latest available version. After the first installation, you can upgrade Fluent Bit as you would normally do with any Linux package.

You can list the available Fluent Bit versions by running:

RPM:

```bash
sudo yum check-update
yum list fluent-bit --showduplicates
```

DEB:

```bash
sudo apt update
apt-cache showpkg fluent-bit
```

To upgrade to the latest Fluent Bit version, run these commands:

RPM:

```bash
# Remove command only required when downgrading to a previous version
# sudo yum remove fluent-bit
sudo yum install fluent-bit
```

DEB:

```bash
sudo apt install fluent-bit
```
Please note that td-agent-bit is not available for the following distributions so rollbacks on them are not possible:
- CentOS 9 Stream (including Rocky Linux and AlmaLinux)
- RedHat 9
- Ubuntu 22.04.x
- Open Suse (SLES) 15.4
- Amazon Linux 2023


If you wish to revert to td-agent-bit, you can follow the steps outlined below:

1. Open the file `/etc/newrelic-infra.yml` using your preferred text editor.
2. Add the following line at the end of the file: `fluent_bit_exe_path: /opt/td-agent-bit/bin/td-agent-bit`.
3. Save the changes.
4. Restart the infrastructure agent by executing the following command: `sudo systemctl restart newrelic-infra`.


By completing these steps, the infrastructure agent will be configured to use td-agent-bit instead of fluent-bit.

What's next? [#what-next]

Explore logging data across your platform with our logs UI.

Disable log forwarding [#uninstall]

To disable log forwarding capabilities, go to your logging.d directory, and remove files with the .yml extension that were originally added during the configuration process.

  • Linux: /etc/newrelic-infra/logging.d/
  • Windows: C:\Program Files\New Relic\newrelic-infra\logging.d\