Splunk HTTP Event Collector (HEC) Exporter

Status
Stability	beta: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@atoulme, @dmitryax

Configuration

The following configuration options are required:

• token (no default): HEC requires a token to authenticate incoming traffic. To procure a token, please refer to the Splunk documentation.
• endpoint (no default): Splunk HEC URL.

The following configuration options can also be configured:

• source (no default): Optional Splunk source: https://docs.splunk.com/Splexicon:Source
• sourcetype (no default): Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
• index (no default): Splunk index, optional name of the Splunk index targeted
• use_multi_metric_format (default: false): Combines metrics with the same metadata to reduce ingest using the multiple-metric JSON format. Applicable in the metrics pipeline only.
• disable_compression (default: false): Whether to disable gzip compression over HTTP.
• timeout (default: 10s): HTTP timeout when sending data.
• http2_read_idle_timeout (default = 10s): Send a ping frame for a health check if the connection has been idle for the configured value. 0s means http/2 health check will be disabled.
• http2_ping_timeout (default = 10s): Triggered by http2_read_idle_timeout; When there's no response to the ping within the configured value, the connection will be closed. If this value is set to 0, it will default to 15s.
• insecure_skip_verify (default: false): Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS.
• ca_file (no default) Path to the CA cert to verify the server being connected to.
• cert_file (no default) Path to the TLS cert to use for client connections when TLS client auth is required.
• key_file (no default) Path to the TLS key to use for TLS required connections.
• max_content_length_logs (default: 2097152): Maximum log payload size in bytes. Log batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only 1 request per batch.
• max_content_length_metrics (default: 2097152): Maximum metric payload size in bytes. Metric batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only one request per batch.
• max_content_length_traces (default: 2097152): Maximum trace payload size in bytes. Trace batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only one request per batch.
• max_event_size (default: 5242880): Maximum raw uncompressed individual event size in bytes. Maximum allowed value is 838860800 (~ 800 MB).
• splunk_app_name (default: "OpenTelemetry Collector Contrib") App name is used to track telemetry information for Splunk App's using HEC by App name.
• splunk_app_version (default: Current OpenTelemetry Collector Contrib Build Version): App version is used to track telemetry information for Splunk App's using HEC by App version.
• log_data_enabled (default: true): Specifies whether the log data is exported. Set it to false if you want the log data to be dropped instead. Applicable in the logs pipeline only.
• profiling_data_enabled (default: true): Specifies whether the profiling data is exported. Set it to false if you want the profiling data to be dropped instead. Applicable in the logs pipeline only.
• health_path (default = '/services/collector/health'): The path reporting health checks.
• health_check_enabled (default = false): Whether to perform Splunk HEC Health Check during the exporter's startup.
• export_raw (default = false): send only the log's body, targeting a Splunk HEC raw endpoint.
• otel_attrs_to_hec_metadata/source (default = 'com.splunk.source'): Specifies the mapping of a specific unified model attribute value to the standard source field of a HEC event.
• otel_attrs_to_hec_metadata/sourcetype (default = 'com.splunk.sourcetype'): Specifies the mapping of a specific unified model attribute value to the standard sourcetype field of a HEC event.
• otel_attrs_to_hec_metadata/index (default = 'com.splunk.index'): Specifies the mapping of a specific unified model attribute value to the standard index field of a HEC event.
• otel_attrs_to_hec_metadata/host (default = 'host.name'): Specifies the mapping of a specific unified model attribute value to the standard host field and the host.name field of a HEC event.
• hec_metadata_to_otel_attrs/source (default = 'com.splunk.source'): Specifies the mapping of a specific unified model attribute value to the standard source field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/source.
• hec_metadata_to_otel_attrs/sourcetype (default = 'com.splunk.sourcetype'): Specifies the mapping of a specific unified model attribute value to the standard sourcetype field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/sourcetype.
• hec_metadata_to_otel_attrs/index (default = 'com.splunk.index'): Specifies the mapping of a specific unified model attribute value to the standard index field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/index.
• hec_metadata_to_otel_attrs/host (default = 'host.name'): Specifies the mapping of a specific unified model attribute value to the standard host field and the host.name field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/host.
• otel_to_hec_fields/severity_text (default = otel.log.severity.text): Specifies the name of the field to map the severity text field of log events.
• otel_to_hec_fields/severity_number (default = otel.log.severity.number): Specifies the name of the field to map the severity number field of log events.
• otel_to_hec_fields/name (default = "otel.log.name): Specifies the name of the field to map the name field of log events.
• heartbeat/interval (no default): Specifies the interval of sending hec heartbeat to the destination. If not specified, heartbeat is not enabled.
• heartbeat/startup (default: false): Check heartbeat at start up time. This action enforces a synchronous heartbeat action during the collector start up sequence. The collector will fail to start if the heartbeat returns an error.
• telemetry/enabled (default: false): Specifies whether to enable telemetry inside splunk hec exporter.
• telemetry/override_metrics_names (default: empty map): Specifies the metrics name to overrides in splunk hec exporter.
• telemetry/extra_attributes (default: empty map): Specifies the extra metrics attributes in splunk hec exporter.
• batcher(Experimental, disabled by default): Specifies batching configuration on the exporter.

In addition, this exporter offers queued retry which is enabled by default. For more information, see the queued retry options in the exporter documentation.
If you are getting throttled due to high volume of events the collector might experience memory issues, in those cases it is recommended to change the queued retry configuration to drop events more frequently, for example you can reduce the maximum amount of time spent trying to send a batch from 120s (default) to 60s:

exporters:
  splunk_hec:
    retry_on_failure:
      max_elapsed_time: 60

If that does not resolve the memory issues you can try to reduce it further and adjust the other queued retry parameters accordingly.
As a last resort after you have tried to solve the memory issues by adjusting the queued retry configuration you can disable it altogether:

exporters:
  splunk_hec:
    retry_on_failure:
      enabled: false

Example:

exporters:
  splunk_hec:
    # Splunk HTTP Event Collector token.
    token: "00000000-0000-0000-0000-0000000000000"
    # URL to a Splunk instance to send data to.
    endpoint: "https://splunk:8088/services/collector"
    # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
    source: "otel"
    # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "otel"
    # Splunk index, optional name of the Splunk index targeted.
    index: "metrics"
    # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
    max_idle_conns: 200
    # Whether to disable gzip compression over HTTP. Defaults to false.
    disable_compression: false
    # HTTP timeout when sending data. Defaults to 10s.
    timeout: 10s
    tls:
      # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
      insecure_skip_verify: false
      # Path to the CA cert to verify the server being connected to.
      ca_file: /certs/ExampleCA.crt
      # Path to the TLS cert to use for client connections when TLS client auth is required.
      cert_file: /certs/HECclient.crt
      # Path to the TLS key to use for TLS required connections.
      key_file: /certs/HECclient.key
    # Application name is used to track telemetry information for Splunk App's using HEC by App name.
    splunk_app_name: "OpenTelemetry-Collector Splunk Exporter"
    # Application version is used to track telemetry information for Splunk App's using HEC by App version.
    splunk_app_version: "v0.0.1"
    heartbeat:
      interval: 30s
    telemetry:
      enabled: true
      override_metrics_names:
        otelcol_exporter_splunkhec_heartbeats_sent: app_heartbeats_success_total
        otelcol_exporter_splunkhec_heartbeats_failed: app_heartbeats_failed_total
      extra_attributes:
        dataset_name: SplunkCloudBeaverStack
        custom_key: custom_value

The full list of settings exposed for this exporter are documented in config.go with detailed sample configurations in testdata/config.yaml.

This exporter also offers proxy support.

Advanced Configuration

Several helper files are leveraged to provide additional capabilities automatically:

• HTTP settings
• TLS and mTLS settings
• Queuing, retry and timeout settings