Pulsar Exporter

Status
Stability	alpha: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@dmitryax, @dao-jun

Pulsar exporter exports logs, metrics, and traces to Apache Pulsar. This exporter uses a synchronous producer that blocks and able to batch messages.

Get Started

The following settings can be optionally configured:

• endpoint (default = pulsar://localhost:6650): The url of pulsar cluster.
• topic (default = otlp_spans for traces, otlp_metrics for metrics, otlp_logs for logs): The name of the pulsar topic to export to.
• encoding (default = otlp_proto): The encoding of the traces sent to pulsar. All available encodings:
  • otlp_proto: payload is Protobuf serialized from ExportTraceServiceRequest if set as a traces exporter or ExportMetricsServiceRequest for metrics or ExportLogsServiceRequest for logs.
  • otlp_json: ** EXPERIMENTAL ** payload is JSON serialized from ExportTraceServiceRequest if set as a traces exporter or ExportMetricsServiceRequest for metrics or ExportLogsServiceRequest for logs.
  • The following encodings are valid only for traces.
    • jaeger_proto: the payload is serialized to a single Jaeger proto Span, and keyed by TraceID.
    • jaeger_json: the payload is serialized to a single Jaeger JSON Span using jsonpb, and keyed by TraceID.
• auth
  • tls
    • cert_file:
    • key_file:
  • token
    • token
  • oauth2
    • issuer_url:
    • client_id:
    • audience:
  • athenz
    • provider_domain:
    • tenant_domain:
    • tenant_service:
    • private_key:
    • key_id:
    • principal_header:
    • zts_url:
• producer
  • max_reconnect_broker: specifies the maximum retry number of reconnectToBroker. (default: ultimate)
  • hashing_scheme: used to define the partition on where to publish a particular message. Can be set to java_string_hash (default) or murmur3_32hash.
  • compression_level: one of 'default' (default), 'faster', or 'better'.
  • compression_type: one of 'none' (default), 'lz4', 'zlib', or 'zstd'.
  • max_pending_messages: specifies the max size of the queue holding the messages pending to receive an acknowledgment from the broker.
  • batch_builder_type: one of 'default' (default) or 'key_based'.
  • partitions_auto_discovery_interval: the time interval for the background process to discover new partitions
  • batching_max_publish_delay: specifies the time period within which the messages sent will be batched (default: 10ms)
  • batching_max_messages: specifies the maximum number of messages permitted in a batch. (default: 1000)
  • batching_max_size: specifies the maximum number of bytes permitted in a batch. (default 128 KB)
  • disable_block_if_queue_full: controls whether Send and SendAsync block if producer's message queue is full. Defaults to false.
  • disable_batching: controls whether automatic batching of messages is enabled for the producer. Defaults to false.
• tls_trust_certs_file_path: path to the CA cert. For a client this verifies the server certificate. Should only be used if insecure is set to true.
• tls_allow_insecure_connection: configure whether the Pulsar client accept untrusted TLS certificate from broker (default: false)
• timeout: send pulsar message timeout (default: 5s)
• operation_timeout: sets producer-create, subscribe and unsubscribe operations timeout (default: 30 seconds)
• connection_timeout: timeout for the establishment of a TCP connection (default: 5 seconds)
• max_connections_per_broker: max number of connections to a single broker that will kept in the pool. (default: 1 connection)
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false
• sending_queue
  • enabled (default = true)
  • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
  • queue_size (default = 1000): Maximum number of batches kept in memory before dropping data; ignored if enabled is false; User should calculate this as num_seconds * requests_per_second where:
    • num_seconds is the number of seconds to buffer in case of a backend outage
    • requests_per_second is the average number of requests per seconds.

Example configuration:

exporters:
  pulsar:
    endpoint: pulsar://localhost:6650
    topic: otlp-spans
    encoding: otlp_proto
    auth:
      tls:
        cert_file: cert.pem
        key_file: key.pem
    timeout: 10s
    tls_allow_insecure_connection: false
    tls_trust_certs_file_path: ca.pem