Google Pubsub Receiver

Status
Stability	beta: traces, logs, metrics
Distributions	contrib
Issues
Code Owners	@alexvanboxel

⚠️ This is a community-provided module. It has been developed and extensively tested at Collibra, but it is not officially supported by GCP.

This receiver gets OTLP messages from a Google Cloud Pubsub subscription.

The following configuration options are supported:

• project (Optional): The Google Cloud Project of the client connects to.
• subscription (Required): The subscription name to receive OTLP data from. The subscription name should be a fully qualified resource name (eg: projects/otel-project/subscriptions/otlp).
• encoding (Optional): The encoding that will be used to received data from the subscription. This can either be otlp_proto_trace, otlp_proto_metric, otlp_proto_log, cloud_logging, or raw_text (see encoding). This will only be used as a fallback, when no content-type attribute is present.
• compression (Optional): The compression that will be used on received data from the subscription. When set it can only be gzip. This will only be used as a fallback, when no content-encoding attribute is present.
• endpoint (Optional): Override the default Pubsub Endpoint, useful when connecting to the PubSub emulator instance or switching between global and regional service endpoints.
• insecure (Optional): allows performing “insecure” SSL connections and transfers, useful when connecting to a local emulator instance. Only has effect if Endpoint is not ""

receivers:
  googlecloudpubsub:
    project: otel-project
    subscription: projects/otel-project/subscriptions/otlp-logs
    encoding: raw_json

Encoding

The encoding options allows you to specify Encoding Extensions for decoding messages on the subscription. An extension need to be configured in the extensions section, and added to pipeline in the collectors configuration file.

The following example shows how to use the text encoding extension for ingesting arbitrary text message on a subscription, wrapping them in OTLP Log messages. Note that not all extensions support all signals.

extensions:
  text_encoding:
    encoding: utf8
    unmarshaling_separator: "\r?\n"

service:
  extensions: [text_encoding]
  pipelines:
    logs:
      receivers: [googlecloudpubsub]
      processors: []
      exporters: [debug]

The receiver also supports build in encodings for the native OTLP encodings, without the need to specify an Encoding Extensions. The non OTLP build in encodings will be deprecated as soon as extensions for the formats are available.

encoding	description
otlp_proto_trace	Decode OTLP trace message
otlp_proto_metric	Decode OTLP trace message
otlp_proto_log	Decode OTLP trace message
cloud_logging	Decode Cloud Logging LogEntry message type
raw_text	Wrap in an OTLP log message

With cloud_logging, the receiver can be used to bring Cloud Logging messages into an OpenTelemetry pipeline. You'll first need to set up a logging sink with a Pub/Sub topic as its destination. Note that the cloud_logging integration is considered alpha as the semantic convention on some of the conversion are not stabilized yet.

With raw_text, the receiver can be used for ingesting arbitrary text message on a Pubsub subscription, wrapping them in OTLP Log messages.

When no encoding is specified, the receiver will try to discover the type of the data by looking at the ce-type and content-type attributes of the message. These message attributes are set by the googlepubsubexporter.

ce-type	ce-datacontenttype	encoding	description
org.opentelemetry.otlp.traces.v1	application/protobuf		Decode OTLP trace message
org.opentelemetry.otlp.metrics.v1	application/protobuf		Decode OTLP metric message
org.opentelemetry.otlp.logs.v1	application/protobuf		Decode OTLP log message

Pubsub subscription

The Google Cloud Pubsub receiver doesn't automatically create subscriptions, it expects the subscription to be created upfront. Security wise it's best to give the collector its own service account and give the subscription Pub/Sub Subscriber permission.

The subscription should also be of delivery type Pull.

Filtering

When the messages on the subscription are accompanied by the correct attributes and you only need a specific type in your pipeline, the messages can be filtered on the subscription saving on egress fees.

An example of filtering on trace message only:

attributes.ce-type = "org.opentelemetry.otlp.traces.v1"
AND
attributes.content-type = "application/protobuf"