Skip to content

mdoi/fluent-plugin-gcloud-pubsub

Repository files navigation

fluent-plugin-gcloud-pubsub

Build Status Gem Version

Overview

Cloud Pub/Sub Input/Output plugin for Fluentd with gcloud gem

  • Publish BufferedOutput chunk into Cloud Pub/Sub with batch publishing
  • Pull messages from Cloud Pub/Sub

Preparation

  • Create a project on Google Developer Console
  • Add a topic of Cloud Pub/Sub to the project
  • Add a pull style subscription to the topic
  • Download your credential (json)

Publish messages

Use out_gcloud_pubsub.

Configuration

publish dummy json data like {"message": "dummy", "value": 0}\n{"message": "dummy", "value": 1}\n ....

<source>
  type dummy
  tag example.publish
  rate 100 
  auto_increment_key value
</source>

<match example.publish>
  type gcloud_pubsub
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  key <YOUR KEY>
  flush_interval 10
  autocreate_topic false
</match>
  • autocreate_topic (optional, default: false)
    • If set to true, specified topic will be created when it doesn't exist.

Pull messages

Use in_gcloud_pubsub.

Configuration

Pull json data from Cloud Pub/Sub

<source>
  type gcloud_pubsub
  tag example.pull
  project <YOUR PROJECT>
  topic <YOUR TOPIC>
  subscription <YOUR SUBSCRIPTION>
  key <YOUR KEY>
  max_messages 1000
  return_immediately true
  pull_interval 2
  format json
</source>

<match example.pull>
  type stdout
</match>

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages