fix: Updated Loki Otlp Ingest Configuration (#12648)

Co-authored-by: J Stickler <julie.stickler@grafana.com>
grafana · Apr 18, 2024 · ff88f3c · ff88f3c
1 parent 0d8ff9e
commit ff88f3c
Showing 1 changed file with 72 additions and 61 deletions.
diff --git a/docs/sources/send-data/otel/_index.md b/docs/sources/send-data/otel/_index.md
@@ -16,14 +16,21 @@ For ingesting logs to Loki using the OpenTelemetry Collector, you must use the [
 
 When logs are ingested by Loki using an OpenTelemetry protocol (OTLP) ingestion endpoint, some of the data is stored as [Structured Metadata]({{< relref "../../get-started/labels/structured-metadata" >}}).
 
+You must set `allow_structured_metadata` to `true` within your Loki config file. Otherwise, Loki will reject the log payload as malformed.
+
+```yaml
+limits_config:
+  allow_structured_metadata: true
+```
+
 ## Configure the OpenTelemetry Collector to write logs into Loki
 
 You need to make the following changes to the [OpenTelemetry Collector config](https://opentelemetry.io/docs/collector/configuration/) to write logs to Loki on its OTLP ingestion endpoint.
 
 ```yaml
 exporters:
   otlphttp:
-    endpoint: http://<loki-addr>/otlp
+    endpoint: http://<loki-addr>:3100/otlp
 ```
 
 And enable it in `service.pipelines`:
@@ -50,7 +57,7 @@ exporters:
   otlphttp:
     auth:
       authenticator: basicauth/otlp
-    endpoint: http://<loki-addr>/otlp
+    endpoint: http://<loki-addr>:3100/otlp
 
 service:
   extensions: [basicauth/otlp]
@@ -118,50 +125,52 @@ It currently only supports changing the storage of Attributes. Here is how the c
 
 ```yaml
 # OTLP log ingestion configurations
-otlp_config:
-  # Configuration for Resource Attributes to store them as index labels or
-  # Structured Metadata or drop them altogether
-  resource_attributes:
-    # Configure whether to ignore the default list of resource attributes set in
-    # 'distributor.otlp.default_resource_attributes_as_index_labels' to be
-    # stored as index labels and only use the given resource attributes config
-    [ignore_defaults: <boolean>]
-
-    [attributes_config: <list of attributes_configs>]
-
-  # Configuration for Scope Attributes to store them as Structured Metadata or
-  # drop them altogether
-  [scope_attributes: <list of attributes_configs>]
-
-  # Configuration for Log Attributes to store them as Structured Metadata or
-  # drop them altogether
-  [log_attributes: <list of attributes_configs>]
-
-attributes_config:
-  # Configures action to take on matching Attributes. It allows one of
-  # [structured_metadata, drop] for all Attribute types. It additionally allows
-  # index_label action for Resource Attributes
-  [action: <string> | default = ""]
-
-  # List of attributes to configure how to store them or drop them altogether
-  [attributes: <list of strings>]
-
-  # Regex to choose attributes to configure how to store them or drop them
-  # altogether
-  [regex: <Regexp>]
+limits_config:
+  otlp_config:
+    # Configuration for Resource Attributes to store them as index labels or
+    # Structured Metadata or drop them altogether
+    resource_attributes:
+      # Configure whether to ignore the default list of resource attributes set in
+      # 'distributor.otlp.default_resource_attributes_as_index_labels' to be
+      # stored as index labels and only use the given resource attributes config
+      [ignore_defaults: <boolean>]
+
+      [attributes_config: <list of attributes_configs>]
+
+    # Configuration for Scope Attributes to store them as Structured Metadata or
+    # drop them altogether
+    [scope_attributes: <list of attributes_configs>]
+
+    # Configuration for Log Attributes to store them as Structured Metadata or
+    # drop them altogether
+    [log_attributes: <list of attributes_configs>]
+
+  attributes_config:
+    # Configures action to take on matching Attributes. It allows one of
+    # [structured_metadata, drop] for all Attribute types. It additionally allows
+    # index_label action for Resource Attributes
+    [action: <string> | default = ""]
+
+    # List of attributes to configure how to store them or drop them altogether
+    [attributes: <list of strings>]
+
+    # Regex to choose attributes to configure how to store them or drop them
+    # altogether
+    [regex: <Regexp>]
 ```
 
 Here are some example configs to change the default mapping of OTLP to Loki format:
 
 #### Example 1:
 
 ```yaml
-otlp_config:
-  resource_attributes:
-    attributes_config:
-      - action: index_label
-        attributes:
-          - service.group
+limits_config:
+  otlp_config:
+    resource_attributes:
+      attributes_config:
+        - action: index_label
+          attributes:
+            - service.group
 ```
 
 With the example config, here is how various kinds of Attributes would be stored:
@@ -172,12 +181,13 @@ With the example config, here is how various kinds of Attributes would be stored
 #### Example 2:
 
 ```yaml
-otlp_config:
-  resource_attributes:
-    ignore_defaults: true
-    attributes_config:
-      - action: index_label
-        regex: service.group
+limits_config:
+  otlp_config:
+    resource_attributes:
+      ignore_defaults: true
+      attributes_config:
+        - action: index_label
+          regex: service.group
 ```
 
 With the example config, here is how various kinds of Attributes would be stored:
@@ -188,25 +198,26 @@ With the example config, here is how various kinds of Attributes would be stored
 #### Example 2:
 
 ```yaml
-otlp_config:
-  resource_attributes:
-    attributes_config:
-      - action: index_label
-        regex: service.group
-  scope_attributes:
-    - action: drop
-      attributes:
-        - method.name
-  log_attributes:
-    - action: structured_metadata
-      attributes:
-        - user.id
-    - action: drop
-      regex: .*
+limits_config:
+  otlp_config:
+    resource_attributes:
+      attributes_config:
+        - action: index_label
+          regex: service.group
+    scope_attributes:
+      - action: drop
+        attributes:
+          - method.name
+    log_attributes:
+      - action: structured_metadata
+        attributes:
+          - user.id
+      - action: drop
+        regex: .*
 ```
 
 With the example config, here is how various kinds of Attributes would be stored:
 * Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
 * Store remaining Resource Attributes as Structured Metadata.
 * Drop Scope Attribute named `method.name` and store all other Scope Attributes as Structured Metadata.
-* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.
+* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.