Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions content/en/agent/logs/log_transport.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ further_reading:
---


## Default agent behavior
## Default Agent behavior

For Agent v6.19+/v7.19+, the default transport used for your logs is compressed HTTPS instead of TCP for the previous versions.
When the Agent starts, if log collection is enabled, it runs a HTTPS connectivity test. If successful, then the Agent uses the compressed HTTPS transport, otherwise the Agent falls back to a TCP transport.
Expand All @@ -33,7 +33,7 @@ To check which transport is used by the Agent, run the [Agent status command][1]

**Notes**:

* For older Agent versions, TCP transport is used by default. Datadog strongly recommends you to enforce HTTPS transport if you are running v6.14+/v7.14+ and HTTPS compression if you are running v6.16+/v7.16+.
* For older Agent versions, TCP transport is used by default. **Datadog strongly recommends** you to enforce HTTPS transport if you are running v6.14+/v7.14+ and HTTPS compression if you are running v6.16+/v7.16+.
* Always enforce a specific transport (either TCP or HTTPS) when using a proxy to forwards logs to Datadog

## Enforce a specific transport
Expand Down Expand Up @@ -111,6 +111,9 @@ When logs are sent through HTTPS, use the same [set of proxy settings][3] as the
{{% tab "TCP" %}}
{{< site-region region="us,eu,us3,us5,ap1,ap2" >}}

{{% logs-tcp-disclaimer %}}


To enforce TCP transport, update the Agent's [main configuration file][1] (`datadog.yaml`) with:

```yaml
Expand Down
4 changes: 3 additions & 1 deletion content/en/agent/logs/proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ further_reading:
text: "Collect your traces"
---

{{% site-region region="us3,eu,us5,gov,ap1,ap2" %}}
{{% logs-tcp-disclaimer %}}

{{% site-region region="us3,us5,gov,ap1,ap2" %}}
<div class="alert alert-danger">
TCP is not available for the {{< region-param key="dd_site_name" >}} site. Contact <a href="/help/">support</a> for more information.
</div>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,30 +134,6 @@ If the Logs Agent status looks like the example in [Check the Agent status](#che
* The required port (10516) for sending logs to Datadog is being blocked.
* Your container is using a different logging driver than the Agent expects.

#### Outbound traffic on port 10516 is blocked

The Datadog Agent sends its logs to Datadog over TCP using port 10516. If that connection is not available, logs fail to be sent and an error is recorded in the `agent.log` file to that effect.

You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For OpenSSL, run the following command:

```shell
openssl s_client -connect intake.logs.datadoghq.com:10516
```

For GnuTLS, run the following command:

```shell
gnutls-cli intake.logs.datadoghq.com:10516
```

And then by sending a log like the following:

```text
<API_KEY> this is a test message
```

If opening the port 10516 is not an option, it is possible to configure the Datadog Agent to send logs through HTTPS by setting the `DD_LOGS_CONFIG_FORCE_USE_HTTP` environment variable to `true`:

#### Your containers are not using the JSON logging driver

Docker's default is the json-file logging driver so the Container Agent tries to read from this first. If your containers are set to use a different logging driver, the Logs Agent indicates that it is able to successfully find your containers but it isn't able to collect their logs. The Container Agent can also be configured to read from the journald logging driver.
Expand Down
31 changes: 0 additions & 31 deletions content/en/logs/guide/log-collection-troubleshooting-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,37 +23,6 @@ There are a number of common issues that can get in the way when [sending new lo

Changes in the configuration of the `datadog-agent` won't be taken into account until you have [restarted the Agent][3].

## Outbound traffic on port 10516 is blocked

The Datadog Agent sends its logs to Datadog over TCP using port 10516. If that connection is not available, logs fail to be sent and an error is recorded in the `agent.log` file to that effect.

You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For OpenSSL, run the following command:

```shell
openssl s_client -connect intake.logs.datadoghq.com:10516
```

For GnuTLS, run the following command:

```shell
gnutls-cli intake.logs.datadoghq.com:10516
```

And then by sending a log like the following:

```text
<API_KEY> this is a test message
```

- If opening the port 10516 is not an option, it is possible to configure the Datadog Agent to send logs through HTTPS by adding the following in `datadog.yaml`:

```yaml
logs_config:
force_use_http: true
```

See the [HTTPS log forwarding section][4] for more information.

## Check the status of the Agent

Often, checking the [Agent status command][5] results will help you troubleshoot what is happening.
Expand Down
118 changes: 22 additions & 96 deletions content/en/logs/log_collection/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,11 +154,6 @@ Use the [site][13] selector dropdown on the right side of the page to see suppor
| US | HTTPS | `agent-http-intake.logs.datadoghq.com` | 443 | Used by the Agent to send logs in JSON format over HTTPS. See the [Host Agent Log collection documentation][2]. |
| US | HTTPS | `lambda-http-intake.logs.datadoghq.com` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS. |
| US | HTTPS | `logs.`{{< region-param key="browser_sdk_endpoint_domain" code="true" >}} | 443 | Used by the Browser SDK to send logs in JSON format over HTTPS. |
| US | TCP | `agent-intake.logs.datadoghq.com` | 10514 | Used by the Agent to send logs without TLS.
| US | TCP and TLS | `agent-intake.logs.datadoghq.com` | 10516 | Used by the Agent to send logs with TLS.
| US | TCP and TLS | `intake.logs.datadoghq.com` | 443 | Used by custom forwarders to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |
| US | TCP and TLS | `functions-intake.logs.datadoghq.com` | 443 | Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. **Note**: This endpoint may be useful with other cloud providers. |
| US | TCP and TLS | `lambda-intake.logs.datadoghq.com` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |

[1]: /api/latest/logs/#send-logs
[2]: /agent/logs/#send-logs-over-https
Expand All @@ -172,9 +167,6 @@ Use the [site][13] selector dropdown on the right side of the page to see suppor
| EU | HTTPS | `agent-http-intake.logs.datadoghq.eu` | 443 | Used by the Agent to send logs in JSON format over HTTPS. See the [Host Agent Log collection documentation][2]. |
| EU | HTTPS | `lambda-http-intake.logs.datadoghq.eu` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over HTTPS. |
| EU | HTTPS | `logs.`{{< region-param key="browser_sdk_endpoint_domain" code="true" >}} | 443 | Used by the Browser SDK to send logs in JSON format over HTTPS. |
| EU | TCP and TLS | `agent-intake.logs.datadoghq.eu` | 443 | Used by the Agent to send logs in protobuf format over an SSL-encrypted TCP connection. |
| EU | TCP and TLS | `functions-intake.logs.datadoghq.eu` | 443 | Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. **Note**: This endpoint may be useful with other cloud providers. |
| EU | TCP and TLS | `lambda-intake.logs.datadoghq.eu` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |

[1]: /api/latest/logs/#send-logs
[2]: /agent/logs/#send-logs-over-https
Expand Down Expand Up @@ -252,95 +244,9 @@ Use the [site][13] selector dropdown on the right side of the page to see suppor

### Custom log forwarding

Any custom process or logging library able to forward logs through **TCP** or **HTTP** can be used in conjunction with Datadog Logs.

{{< tabs >}}
{{% tab "HTTP" %}}

You can send logs to Datadog platform over HTTP. Refer to the [Datadog Log HTTP API documentation][1] to get started.

[1]: /api/latest/logs/#send-logs
{{% /tab %}}
{{% tab "TCP" %}}

{{< site-region region="us" >}}

You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For GnuTLS, run the following command:

```shell
gnutls-cli intake.logs.datadoghq.com:10516
```

For OpenSSL, run the following command:

```shell
openssl s_client -connect intake.logs.datadoghq.com:10516
```

You must prefix the log entry with your [Datadog API Key][1] and add a payload.

```
<DATADOG_API_KEY> Log sent directly using TLS
```

Your payload, or `Log sent directly using TLS` as written in the example, can be in raw, Syslog, or JSON format. If your payload is in JSON format, Datadog automatically parses its attributes.

```text
<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}
```

[1]: /account_management/api-app-keys/#api-keys

{{< /site-region >}}

{{< site-region region="eu" >}}

You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For GnuTLS, run the following command:

```shell
gnutls-cli tcp-intake.logs.datadoghq.eu:443
```

For OpenSSL, run the following command:

```shell
openssl s_client -connect tcp-intake.logs.datadoghq.eu:443
```

You must prefix the log entry with your [Datadog API Key][1] and add a payload.
Any custom process or logging library able to forward logs through **HTTP** can be used in conjunction with Datadog Logs.

```
<DATADOG_API_KEY> Log sent directly using TLS
```

Your payload, or `Log sent directly using TLS` as written in the example, can be in raw, Syslog, or JSON format. If your payload is in JSON format, Datadog automatically parses its attributes.

```text
<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}
```

[1]: /account_management/api-app-keys/#api-keys

{{< /site-region >}}

{{< site-region region="us3" >}}
The TCP endpoint is not recommended for this site. Contact [support][1] for more information.

[1]: /help
{{< /site-region >}}

{{< site-region region="gov,us5,ap1,ap2" >}}

The TCP endpoint is not supported for this site.

[1]: /help
{{< /site-region >}}


[1]: https://app.datadoghq.com/organization-settings/api-keys
[2]: https://app.datadoghq.com/logs/livetail
{{% /tab %}}
{{< /tabs >}}
You can send logs to Datadog platform over HTTP. Refer to the [Datadog Log HTTP API documentation][15] to get started.

**Notes**:

Expand All @@ -357,6 +263,24 @@ Log events that do not comply with these limits might be transformed or truncate

There is an additional truncation in fields that applies only to indexed logs: the value is truncated to 75 KiB for the message field and 25 KiB for non-message fields. Datadog still stores the full text, and it remains visible in regular list queries in the Logs Explorer. However, the truncated version will be displayed when performing a grouped query, such as when grouping logs by that truncated field or performing similar operations that display that specific field.

{{% collapse-content title="TCP" level="h3" expanded=false %}}

{{% logs-tcp-disclaimer %}}


| Site | Type | Endpoint | Port | Description |
|------|-------------|---------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| US | TCP | `agent-intake.logs.datadoghq.com` | 10514 | Used by the Agent to send logs without TLS.
| US | TCP and TLS | `agent-intake.logs.datadoghq.com` | 10516 | Used by the Agent to send logs with TLS.
| US | TCP and TLS | `intake.logs.datadoghq.com` | 443 | Used by custom forwarders to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |
| US | TCP and TLS | `functions-intake.logs.datadoghq.com` | 443 | Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. **Note**: This endpoint may be useful with other cloud providers. |
| US | TCP and TLS | `lambda-intake.logs.datadoghq.com` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |
| EU | TCP and TLS | `agent-intake.logs.datadoghq.eu` | 443 | Used by the Agent to send logs in protobuf format over an SSL-encrypted TCP connection. |
| EU | TCP and TLS | `functions-intake.logs.datadoghq.eu` | 443 | Used by Azure functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. **Note**: This endpoint may be useful with other cloud providers. |
| EU | TCP and TLS | `lambda-intake.logs.datadoghq.eu` | 443 | Used by Lambda functions to send logs in raw, Syslog, or JSON format over an SSL-encrypted TCP connection. |

{{% /collapse-content %}}

### Attributes and tags

Attributes prescribe [logs facets][9], which are used for filtering and searching in Log Explorer. See the dedicated [attributes and aliasing][10] documentation for a list of reserved and standard attributes and to learn how to support a naming convention with logs attributes and aliasing.
Expand Down Expand Up @@ -407,3 +331,5 @@ Once logs are collected and ingested, they are available in **Log Explorer**. Lo
[12]: /logs/explore/
[13]: /getting_started/site/
[14]: /logs/log_configuration/pipelines/?tab=date#date-attribute
[15]: /api/latest/logs/#send-logs

52 changes: 0 additions & 52 deletions content/en/logs/log_collection/csharp.md
Original file line number Diff line number Diff line change
Expand Up @@ -494,58 +494,6 @@ using (var log = new LoggerConfiguration()
}
```

{{< site-region region="us" >}}

You can also override the default behavior and forward logs in TCP by manually specifying the following required properties: `url`, `port`, `useSSL`, and `useTCP`. Optionally, [specify the `source`, `service`, `host`, and custom tags.][1]

For instance to forward logs to the Datadog US region in TCP you would use the following sink configuration:

```csharp
var config = new DatadogConfiguration(url: "intake.logs.datadoghq.com", port: 10516, useSSL: true, useTCP: true);
using (var log = new LoggerConfiguration()
.WriteTo.DatadogLogs(
"<API_KEY>",
source: "<SOURCE_NAME>",
service: "<SERVICE_NAME>",
host: "<HOST_NAME>",
tags: new string[] {"<TAG_1>:<VALUE_1>", "<TAG_2>:<VALUE_2>"},
configuration: config
)
.CreateLogger())
{
// Some code
}
```

[1]: /logs/log_configuration/attributes_naming_convention/#reserved-attributes

{{< /site-region >}}
{{< site-region region="eu" >}}

You can also override the default behavior and forward logs in TCP by manually specifying the following required properties: `url`, `port`, `useSSL`, and `useTCP`. Optionally, [specify the `source`, `service`, `host`, and custom tags.][1]

For instance to forward logs to the Datadog EU region in TCP you would use the following sink configuration:

```csharp
var config = new DatadogConfiguration(url: "tcp-intake.logs.datadoghq.eu", port: 443, useSSL: true, useTCP: true);
using (var log = new LoggerConfiguration()
.WriteTo.DatadogLogs(
"<API_KEY>",
source: "<SOURCE_NAME>",
service: "<SERVICE_NAME>",
host: "<HOST_NAME>",
tags: new string[] {"<TAG_1>:<VALUE_1>", "<TAG_2>:<VALUE_2>"},
configuration: config
)
.CreateLogger())
{
// Some code
}
```
[1]: /logs/log_configuration/attributes_naming_convention/#reserved-attributes

{{< /site-region >}}

New logs are now directly sent to Datadog.

## Further Reading
Expand Down
2 changes: 2 additions & 0 deletions layouts/shortcodes/logs-tcp-disclaimer.en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
<div class="alert alert-warning">TCP log collection is <strong>not supported</strong>. Datadog provides <strong>no delivery or reliability guarantees</strong> when using TCP, and log data may be lost without notice.
For reliable ingestion, use the HTTP intake endpoint, an official Datadog Agent, or forwarder integration instead. For more information, see <a href="/logs/log_collection/?tab=host">Log Collection</a>.</div>
Loading