terraform-google-pubsub-subscription

A Terraform module for creating a PubSub Subscriber for Google Cloud Pub/Sub.

This module supports Terraform version 1 and is compatible with the Terraform Google Provider version 5.

This module is part of our Infrastructure as Code (IaC) framework that enables our users and customers to easily deploy and manage reusable, secure, and production-grade cloud infrastructure.

• Module Features
• Getting Started
• Module Argument Reference
  • Main Resource Configuration
  • Extended Resource Configuration
  • Module Configuration
• Module Outputs
• External Documentation
  • Google Documentation
  • Terraform GCP Provider Documentation
• Module Versioning
  • Backwards compatibility in 0.0.z and 0.y.z version
• About Mineiros
• Reporting Issues
• Contributing
• Makefile Targets
• License

Module Features

This module implements the following Terraform resources

• google_pubsub_subscription

and supports additional features of the following modules:

• mineiros-io/subscription-iam

Getting Started

Most common usage of the module:

module "terraform-google-pubsub-subscription" {
  source = "git@github.com:mineiros-io/terraform-google-pubsub-subscription.git?ref=v0.1.0"

  name    = "subscription-name"
  topic   = "topic-resource"
  project = "resource-project"
}

Module Argument Reference

See variables.tf and examples/ for details and use-cases.

name = "name-of-sub" topic = "topic-resource" project = "project-a"

Main Resource Configuration

• name: (Required string)

  Name of the subscription.

• topic: (Required string)

  A reference to a Topic resource.

• project: (Required string)

  The project in which the resource belongs. If it is not provided, the provider project is used.

• labels: (Optional map(string))

  A set of key/value label pairs to assign to this Subscription.

  Default is {}.

• ack_deadline_seconds: (Optional number)

  This value is the maximum time after a subscriber receives a message before the subscriber should acknowledge the message. After message delivery but before the ack deadline expires and before the message is acknowledged, it is an outstanding message and will not be delivered again during that time (on a best-effort basis). For pull subscriptions, this value is used as the initial value for the ack deadline. To override this value for a given message, call subscriptions.modifyAckDeadline with the corresponding ackId if using pull. The minimum custom deadline you can specify is 10 seconds. The maximum custom deadline you can specify is 600 seconds (10 minutes). If this parameter is 0, a default value of 10 seconds is used. For push delivery, this value is also used to set the request timeout for the call to the push endpoint. If the subscriber never acknowledges the message, the Pub/Sub system will eventually redeliver the message.

• message_retention_duration: (Optional string)

  How long to retain unacknowledged messages in the subscription's backlog, from the moment a message is published. If retainAckedMessages is true, then this also configures the retention of acknowledged messages, and thus configures how far back in time a subscriptions.seek can be done. Defaults to 7 days. Cannot be more than 7 days (604800s) or less than 10 minutes (600s). A duration in seconds with up to nine fractional digits, terminated by s.

  Default is "604800s".

  Example:

  600.5s

• retain_acked_messages: (Optional bool)

  Indicates whether to retain acknowledged messages. If true, then messages are not expunged from the subscription's backlog, even if they are acknowledged, until they fall out of the messageRetentionDuration window.

• filter: (Optional string)

  The subscription only delivers the messages that match the filter. Pub/Sub automatically acknowledges the messages that don't match the filter. You can filter messages by their attributes. The maximum length of a filter is 256 bytes. After creating the subscription, you can't modify the filter.

• enable_message_ordering: (Optional bool)

  If true, messages published with the same orderingKey in PubsubMessage will be delivered to the subscribers in the order in which they are received by the Pub/Sub system. Otherwise, they may be delivered in any order.

• enable_exactly_once_delivery: (Optional bool)

  If true,The message sent to a subscriber is guaranteed not to be resent before the message's acknowledgement deadline expires. An acknowledged message will not be resent to a subscriber. Note that subscribers may still receive multiple copies of a message when enable_exactly_once_delivery is true if the message was published multiple times by a publisher client. These copies are considered distinct by Pub/Sub and have distinct messageId values

• expiration_policy_ttl: (Optional string)

  A policy that specifies the conditions for this subscription's expiration. A subscription is considered active as long as any connected subscriber is successfully consuming messages from the subscription or is issuing operations on the subscription. If expirationPolicy is not set, a default policy with ttl of 31 days will be used. If it is set but ttl is "", the resource never expires. The minimum allowed value for expirationPolicy.ttl is 1 day. A duration in seconds with up to nine fractional digits, terminated by 's'.

• dead_letter_policy: (Optional object(dead_letter_policy))

  A policy that specifies the conditions for dead lettering messages in this subscription. If dead_letter_policy is not set, dead lettering is disabled. The Cloud Pub/Sub service account associated with this subscription's parent project (i.e., service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com) must have permission to Acknowledge() messages on this subscription.

  The dead_letter_policy object accepts the following attributes:

  • dead_letter_topic: (Optional string)

    The name of the topic to which dead letter messages should be published. Format is projects/{project}/topics/{topic}. The Cloud Pub/Sub service account associated with the enclosing subscription's parent project (i.e., service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com) must have permission to Publish() to this topic. The operation will fail if the topic does not exist. Users should ensure that there is a subscription attached to this topic since messages published to a topic with no subscriptions are lost.

  • max_delivery_attempts: (Optional number)

    The maximum number of delivery attempts for any message. The value must be between 5 and 100. The number of delivery attempts is defined as 1 + (the sum of number of NACKs and number of times the acknowledgement deadline has been exceeded for the message). A NACK is any call to ModifyAckDeadline with a 0 deadline. Note that client libraries may automatically extend ack_deadlines. This field will be honored on a best effort basis. If this parameter is 0, a default value of 5 is used.

• retry_policy: (Optional object(retry_policy))

  A policy that specifies how Pub/Sub retries message delivery for this subscription. If not set, the default retry policy is applied. This generally implies that messages will be retried as soon as possible for healthy subscribers. RetryPolicy will be triggered on NACKs or acknowledgement deadline exceeded events for a given message.

  The retry_policy object accepts the following attributes:

  • minimum_backoff: (Optional string)

    The minimum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by s.

    Example:

    3.5s

  • maximum_backoff: (Optional string)

    The maximum delay between consecutive deliveries of a given message. Value should be between 0 and 600 seconds. A duration in seconds with up to nine fractional digits, terminated by s.

    Example:

    3.5s

• push_config: (Optional object(push_config))

  If push delivery is used with this subscription, this field is used to configure it. An empty pushConfig signifies that the subscriber will pull and ack messages using API methods.

  The push_config object accepts the following attributes:

  • oidc_token: (Optional object(oidc_token))

    If specified, Pub/Sub will generate and attach an OIDC JWT token as an Authorization header in the HTTP request for every pushed message.

    The oidc_token object accepts the following attributes:

    • service_account_email: (Required string)

      Service account email to be used for generating the OIDC token. The caller (for subscriptions.create, subscriptions.patch, and subscriptions.modifyPushConfig RPCs) must have the iam.serviceAccounts.actAs permission for the service account.

    • audience: (Optional string)

      Audience to be used when generating OIDC token. The audience claim identifies the recipients that the JWT is intended for. The audience value is a single case-sensitive string. Having multiple values (array) for the audience field is not supported. More info about the OIDC JWT token audience here: https://tools.ietf.org/html/rfc7519#section-4.1.3 Note: if not specified, the Push endpoint URL will be used.

  • no_wrapper: (Optional object(no_wrapper))

    When set, the payload to the push endpoint is not wrapped. Sets the data field as the HTTP body for delivery.

    The no_wrapper object accepts the following attributes:

    • write_metadata: (Required string)

      When true, writes the Pub/Sub message metadata to x-goog-pubsub-: headers of the HTTP request. Writes the Pub/Sub message attributes to : headers of the HTTP request.

  • push_endpoint: (Required string)

    A URL locating the endpoint to which messages should be pushed. For example, a Webhook endpoint might use "https://example.com/push".

  • attributes: (Required string)

    Endpoint configuration attributes. Every endpoint has a set of API supported attributes that can be used to control different aspects of the message delivery. The currently supported attribute is x-goog-version, which you can use to change the format of the pushed message. This attribute indicates the version of the data expected by the endpoint. This controls the shape of the pushed message (i.e., its fields and metadata). The endpoint version is based on the version of the Pub/Sub API. If not present during the subscriptions.create call, it will default to the version of the API used to make such call. If not present during a subscriptions.modifyPushConfig call, its value will not be changed. subscriptions.get calls will always return a valid version, even if the subscription was created without this attribute. The possible values for this attribute are:

    • v1beta1: uses the push format defined in the v1beta1 Pub/Sub API.
    • v1 or v1beta2: uses the push format defined in the v1 Pub/Sub API.

• bigquery_config: (Optional object(bigquery_config))

  If delivery to BigQuery is used with this subscription, this field is used to configure it. Either push_config or bigquery_config can be set, but not both. If both are empty, then the subscriber will pull and ack messages using API methods."

  The bigquery_config object accepts the following attributes:

  • table: (Required string)

    The name of the table to which to write data, of the form {projectId}.{datasetId}.{tableId}

  • use_topic_schema: (Optional bool)

    When true, use the topic's schema as the columns to write to in BigQuery, if it exists.

  • write_metadata: (Optional bool)

    When true, write the subscription name, messageId, publishTime, attributes, and orderingKey to additional columns in the table. The subscription name, messageId, and publishTime fields are put in their own columns while all other message properties (other than data) are written to a JSON object in the attributes column.

  • drop_unknown_fields: (Optional bool)

    When true and use_topic_schema is true, any fields that are a part of the topic schema that are not part of the BigQuery table schema are dropped when writing to BigQuery. Otherwise, the schemas must be kept in sync and any messages with extra fields are not written and remain in the subscription's backlog.

• cloud_storage_config: (Optional object(cloud_storage_config))

  If delivery to Cloud Storage is used with this subscription, this field is used to configure it. Either pushConfig, bigQueryConfig or cloudStorageConfig can be set, but not combined. If all three are empty, then the subscriber will pull and ack messages using API methods.

  The cloud_storage_config object accepts the following attributes:

  • bucket: (Required string)

    User-provided name for the Cloud Storage bucket. The bucket must be created by the user. The bucket name must be without any prefix like "gs://".

  • filename_prefix: (Optional string)

    (Optional) User-provided prefix for Cloud Storage filename.

  • filename_suffix: (Optional string)

    (Optional) User-provided suffix for Cloud Storage filename. Must not end in "/".

  • max_duration: (Optional string)

    (Optional) The maximum duration that can elapse before a new Cloud Storage file is created. Min 1 minute, max 10 minutes, default 5 minutes. May not exceed the subscription's acknowledgement deadline. A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

  • max_bytes: (Optional number)

    (Optional) The maximum bytes that can be written to a Cloud Storage file before a new file is created. Min 1 KB, max 10 GiB. The maxBytes limit may be exceeded in cases where messages are larger than the limit.

  • state: (Optional any)

    (Output) An output-only field that indicates whether or not the subscription can receive messages.

  • avro_config: (Optional bool)

    If set, message data will be written to Cloud Storage in Avro format.

    The object accepts the following attributes:

    • write_metadata: (Optional any)

      When true, write the subscription name, messageId, publishTime, attributes, and orderingKey as additional fields in the output.

Extended Resource Configuration

• iam: (Optional list(iam))

  A list of IAM access.

  Default is [].

  Example:

  iam = [{
    role    = "roles/viewer"
    members = ["user:member@example.com"]
  }]

  Each iam object in the list accepts the following attributes:

  • members: (Optional set(string))

    Identities that will be granted the privilege in role. Each entry can have one of the following values:

    • allUsers: A special identifier that represents anyone who is on the internet; with or without a Google account.
    • allAuthenticatedUsers: A special identifier that represents anyone who is authenticated with a Google account or a service account.
    • user:{emailid}: An email address that represents a specific Google account. For example, alice@gmail.com or joe@example.com.
    • serviceAccount:{emailid}: An email address that represents a service account. For example, my-other-app@appspot.gserviceaccount.com.
    • group:{emailid}: An email address that represents a Google group. For example, admins@example.com.
    • domain:{domain}: A G Suite domain (primary, instead of alias) name that represents all the users of that domain. For example, google.com or example.com.
    • computed:{identifier}: An existing key from var.computed_members_map.

    Default is [].

  • role: (Optional string)

    The role that should be applied. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

  • authoritative: (Optional bool)

    Whether to exclusively set (authoritative mode) or add (non-authoritative/additive mode) members to the role.

    Default is true.

• policy_bindings: (Optional list(policy_binding))

  A list of IAM policy bindings.

  Example:

  policy_bindings = [{
    role    = "roles/viewer"
    members = ["user:member@example.com"]
  }]

  Each policy_binding object in the list accepts the following attributes:

  • role: (Required string)

    The role that should be applied.

  • members: (Optional set(string))

    Identities that will be granted the privilege in role.

    Default is var.members.

  • condition: (Optional object(condition))

    An IAM Condition for a given binding.

    Example:

    condition = {
      expression = "request.time < timestamp(\"2022-01-01T00:00:00Z\")"
      title      = "expires_after_2021_12_31"
    }

    The condition object accepts the following attributes:

    • expression: (Required string)

      Textual representation of an expression in Common Expression Language syntax.

    • title: (Required string)

      A title for the expression, i.e. a short string describing its purpose.

    • description: (Optional string)

      An optional description of the expression. This is a longer text which describes the expression, e.g. when hovered over it in a UI.

• computed_members_map: (Optional map(string))

  A map of members to replace in members of various IAM settings to handle terraform computed values.

  Default is {}.

Module Configuration

• module_enabled: (Optional bool)

  Specifies whether resources in the module will be created.

  Default is true.

• module_depends_on: (Optional list(dependency))

  A list of dependencies. Any object can be assigned to this list to define a hidden external dependency.

  Default is [].

  Example:

  module_depends_on = [
    null_resource.name
  ]

Module Outputs

The following attributes are exported in the outputs of the module:

• iam: (list(iam))

  The iam resource objects that define access to the GCS bucket.

• subscription: (object(subscription))

  All attributes of the created subscription.

• module_enabled: (bool)

  Whether this module is enabled.

External Documentation

Google Documentation

• PubSub: https://cloud.google.com/pubsub
• Subscriber Overview: https://cloud.google.com/pubsub/docs/subscriber

Terraform GCP Provider Documentation

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription

Module Versioning

This Module follows the principles of Semantic Versioning (SemVer).

Given a version number MAJOR.MINOR.PATCH, we increment the:

1. MAJOR version when we make incompatible changes,
2. MINOR version when we add functionality in a backwards compatible manner, and
3. PATCH version when we make backwards compatible bug fixes.

Backwards compatibility in 0.0.z and 0.y.z version

• Backwards compatibility in versions 0.0.z is not guaranteed when z is increased. (Initial development)
• Backwards compatibility in versions 0.y.z is not guaranteed when y is increased. (Pre-release)

About Mineiros

Mineiros is a remote-first company headquartered in Berlin, Germany that solves development, automation and security challenges in cloud infrastructure.

Our vision is to massively reduce time and overhead for teams to manage and deploy production-grade and secure cloud infrastructure.

We offer commercial support for all of our modules and encourage you to reach out if you have any questions or need help. Feel free to email us at hello@mineiros.io or join our Community Slack channel.

Reporting Issues

We use GitHub Issues to track community reported issues and missing features.

Contributing

Contributions are always encouraged and welcome! For the process of accepting changes, we use Pull Requests. If you'd like more information, please see our Contribution Guidelines.

Makefile Targets

This repository comes with a handy Makefile. Run make help to see details on each available target.

License

This module is licensed under the Apache License Version 2.0, January 2004. Please see LICENSE for full details.

Copyright © 2020-2022 Mineiros GmbH