Skip to content
Permalink
Browse files
Added Google Pubsub sink connector example
  • Loading branch information
oscerd committed Mar 2, 2021
1 parent c88d25a commit 17f83bc3f3956ee1d853bfe47e89ab67376d9b65
Show file tree
Hide file tree
Showing 2 changed files with 160 additions and 0 deletions.
@@ -0,0 +1,132 @@
= Camel-Kafka-connector Google Pubsub Sink Example

This is an example for Camel-Kafka-connector Google PubSub Sink Source Example

== Standalone

=== What is needed

- A google cloud account
- A service account for PubSub service
- A project in google cloud
- A PubSub topic
- A PubSub topic subscription

=== Setting up PubSub

You need to go here: https://console.cloud.google.com/cloudpubsub
- Create a service account with permission to operate on PubSub service
- Export a service account key file somewhere on your filesystem (for example /home/connectors/ServiceAccountPubsub.json)
- Follow the instruction to create a topic (call it 'test' for example)
- Follow the instruction to create a subscription (call it 'test-sub' for example)

Take note of project id and subscription name. If for example your topic name is 'projects/testpubsub-306306/topics/test'
the project id will be 'testpubsub-306306'.

=== Running Kafka

[source]
----
$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
$KAFKA_HOME/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytopic
----

=== Download the connector package

Download the connector package tar.gz and extract the content to a directory. In this example we'll use `/home/oscerd/connectors/`

[source]
----
> cd /home/oscerd/connectors/
> wget https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.8.0/camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
> untar.gz camel-google-pubsub-kafka-connector-0.8.0-package.tar.gz
----

=== Configuring Kafka Connect

You'll need to set up the `plugin.path` property in your kafka

Open the `$KAFKA_HOME/config/connect-standalone.properties` and set the `plugin.path` property to your choosen location:

[source]
----
...
plugin.path=/home/oscerd/connectors
...
----

=== Setup the connectors

Open the Google Pubsub Sink connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties`

[source]
----
name=CamelGoogle-pubsubSinkConnector
connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
topics=mytopic
camel.sink.path.destinationName=test
camel.sink.path.projectId=testpubsub-306306
camel.sink.endpoint.synchronousPull=true
----

For this example we'll use a Google PubSub Source connector too to get the messages:

Open the Google Pubsub source connector configuration file at `$EXAMPLES/google-pubsub/google-pubsub-source/config/CamelGooglepubsubSourceConnector.properties`

[source]
----
name=CamelGoogle-pubsubSourceConnector
connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
topics=mytopic-source
camel.source.path.destinationName=test-sub
camel.source.path.projectId=testpubsub-306306
camel.source.endpoint.synchronousPull=true
----


=== Running the example

On the same terminal you'll run the connector run the following command:

[source]
----
export GOOGLE_APPLICATION_CREDENTIALS=/home/connectors/ServiceAccountPubsub.json
----

pointing to the service account key you created.

Run the kafka connect with the Google Pubsub Source connector:

[source]
----
$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSinkConnector.properties $EXAMPLES/google-pubsub/google-pubsub-sink/config/CamelGooglepubsubSourceConnector.properties
----

In a terminal with kafkacat run the following command:
[source]
----
> echo "Camel Kafka Connector is great!" | ./kafkacat -b localhost:9092 -t mytopic
% Reached end of topic mytopic [0] at offset 1
----

On a different terminal run the kafkacat consumer

[source]
----
> ./kafkacat -b localhost:9092 -t mytopic-source
Camel Kafka Connector is great!
% Reached end of topic mytopic [0] at offset 1
----

The message in the test topic has been consumed by the source connector now.
@@ -0,0 +1,28 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name=CamelGoogle-pubsubSourceConnector
connector.class=org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSourceConnector
tasks.max=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

topics=mytopic

camel.source.path.destinationName=test-sub
camel.source.path.projectId=testpubsub-306306
camel.source.endpoint.synchronousPull=true

0 comments on commit 17f83bc

Please sign in to comment.