DBZ-654 Instructions for Strimzi 0.2.0 #145

jpechane · 2018-04-11T13:02:35Z

https://issues.jboss.org/browse/DBZ-654

gunnarmorling

@jpechane Thanks a lot! A few comments inline, minor things really.

gunnarmorling · 2018-04-11T13:14:51Z

docs/openshift.asciidoc

@@ -19,30 +19,36 @@ It starts an OpenShift cluster just for you, allowing you to take your first ste
 For setting up Kafka and Kafka Connect on OpenShift, a set of images provided by the http://strimzi.io/[Strimzi] project can be used, which offers "Kafka as a Service".
 It consists of enterprise grade configuration files and images that bring Kafka to OpenShift.

-First we install the Kafka broker and Kafka Connect templates into our OpenShift project:
+First we install the controlloers and templates for the Kafka broker and Kafka Connect into our OpenShift project:


s/controlloers/controllers/

gunnarmorling · 2018-04-11T13:15:41Z

docs/openshift.asciidoc

-oc create -f https://raw.githubusercontent.com/strimzi/strimzi/0.1.0/kafka-statefulsets/resources/openshift-template.yaml
-oc create -f https://raw.githubusercontent.com/strimzi/strimzi/0.1.0/kafka-connect/s2i/resources/openshift-template.yaml
+export STRIMZI_VERSION={strimzi-version}
+git clone -b $STRIMZI_VERSION https://github.com/strimzi/strimzi


I actually liked the fact that it pulled the files via HTTP instead of cloning the repo. Is that not possible any more? Or what did make you change it?

Now it means installing 8 files, check
https://github.com/strimzi/strimzi/tree/0.2.0/examples/templates/cluster-controller
https://github.com/strimzi/strimzi/tree/0.2.0/examples/install/cluster-controller

It seems to me more manageble this way

What not downloading the release from the "releases" and unpacking the tar.gz and then using it for Strimzi ?

WRT the complexity I have no preference for any of those methods - @gunnarmorling you can choose

No strong preference either. Usually I'd prefer getting the distro, but here it doesn't really make a difference as we "just" get some text files, no binaries.

gunnarmorling · 2018-04-11T13:18:55Z

docs/openshift.asciidoc

+# Deploy an ephemeral single instance Kafka broker
+oc new-app strimzi-ephemeral -p ZOOKEEPER_NODE_COUNT=1 -p KAFKA_NODE_COUNT=1 -p KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -p KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
+
+# Deploy a single instance of Kafka Connect instance with no plug-in installed


s/Kafka Connect instance/Kafka Connect/

gunnarmorling · 2018-04-11T13:19:56Z

docs/openshift.asciidoc

 mkdir -p plugins && cd plugins && \
 for PLUGIN in {mongodb,mysql,postgres}; do \
    curl http://central.maven.org/maven2/io/debezium/debezium-connector-$PLUGIN/$DEBEZIUM_VERSION/debezium-connector-$PLUGIN-$DEBEZIUM_VERSION-plugin.tar.gz | tar xz; \
 done && \
-oc start-build debezium --from-dir=. --follow && \
+oc start-build my-connect-cluster-connect --from-dir=. --follow && \


s/my-connect-cluster-connect/my-connect-cluster/? Or perhaps "debezium-kafka-connect"?

gunnarmorling · 2018-04-11T13:25:11Z

docs/openshift.asciidoc

    -H "Accept:application/json" \
    -H "Content-Type:application/json" \
-    http://kafka-connect:8083/connectors -d @- <<'EOF'
+    http://my-connect-cluster-connect:8083/connectors -d @- <<'EOF'


What did make you change the name?

The default name is changed in the new template

Hum, but "my-connect-cluster-connect" really seems odd. Can we make it "my-connect-cluster" at least (ignoring the fact that it isn't really a cluster but just a single node).

jpechane · 2018-04-12T05:51:29Z

@gunnarmorling I tried to use cluster name parameters. The names can't be customized in another way.

gunnarmorling · 2018-04-12T07:53:46Z

That might be interesting feedback for @ppatierno actually. Paolo, we're looking for a way to customize the name of the Kafka Connect cluster set up via Strimzi. Any ideas?

ppatierno · 2018-04-12T08:06:14Z

@gunnarmorling isn't it what you are searching for ?

oc new-app strimzi-connect-s2i -p CLUSTER_NAME=debezium .....

jpechane · 2018-04-12T09:23:05Z

@ppatierno Still the suffix -connect is forced even if you provide your own cluster name

ppatierno · 2018-04-12T10:56:32Z

@jpechane out of curiosity do you have any specific reason and/or use case so that you don't like to have a well defined suffix (the -connect one) after your customizable cluster name ?

jpechane · 2018-04-12T11:07:55Z

@ppatierno connect is too generic - check for product templates namings https://github.com/jboss-openshift/application-templates/blob/master/amq/amq62-basic.json
Also consider chaning CLUSTER_NAME to APPLICATION_NAME to keep things consistent accross products

gunnarmorling · 2018-04-12T13:02:23Z

@jpechane, one more thing: could you try and register the connector with this routing config:

"transforms": "route",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement": "$1_$2_$3"

I see the Connect logs being flooded with lots of messages about the resulting topic name being illegal. Do you observe the same? If so, I'm wondering where this is coming from, as that config worked fine for me when using plain Docker.

gunnarmorling · 2018-04-18T09:21:04Z

@jpechane Did you have a chance to look into that last comment of mine?

jpechane · 2018-04-18T09:50:13Z

@gunnarmorling Sorry, I've commented yesterday in Gitter
I've just tried it and I do not see any problem
Topics were duly created
And records are stored in them
I've used registration request

 {
        "name": "inventory-connector3",
        "config": {
                "connector.class": "io.debezium.connector.mysql.MySqlConnector",
                "tasks.max": "1",
                "database.hostname": "mysql",
                "database.port": "3306",
                "database.user": "debezium",
                "database.password": "dbz",
                "database.server.id": "184054",
                "database.server.name": "dbserver3",
                "database.whitelist": "inventory",
                "database.history.kafka.bootstrap.servers": "broker-kafka:9092",
                "database.history.kafka.topic": "schema-changes.inventory",
                "transforms": "route",
                "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
                "transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
                "transforms.route.replacement": "$1_$2_$3"
        }
}

gunnarmorling · 2018-04-18T18:41:44Z

Weird, ok then. Thanks for checking, no idea what was wrong here then.

Now there's Strimzi 0.3.0 out in between, so could you update the PR accordingly? Thx!

jpechane · 2018-04-19T05:25:13Z

@gunnarmorling Updated and tested

gunnarmorling · 2018-04-19T10:13:41Z

Excellent, thanks, @jpechane! Merging.

DBZ-654 Instructions for Strimzi 0.2.0

ac7fa37

gunnarmorling reviewed Apr 11, 2018

View reviewed changes

DBZ-654 Changed cluster names

0f05ee8

DBZ-654 Strimzi now 0.3.0

8127ebf

gunnarmorling merged commit dd072fc into debezium:develop Apr 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBZ-654 Instructions for Strimzi 0.2.0 #145

DBZ-654 Instructions for Strimzi 0.2.0 #145

jpechane commented Apr 11, 2018

gunnarmorling left a comment

gunnarmorling Apr 11, 2018

gunnarmorling Apr 11, 2018

jpechane Apr 11, 2018

gunnarmorling Apr 11, 2018

ppatierno Apr 12, 2018

jpechane Apr 12, 2018

gunnarmorling Apr 16, 2018

gunnarmorling Apr 11, 2018

gunnarmorling Apr 11, 2018

gunnarmorling Apr 11, 2018

jpechane Apr 11, 2018

gunnarmorling Apr 11, 2018

jpechane commented Apr 12, 2018

gunnarmorling commented Apr 12, 2018

ppatierno commented Apr 12, 2018

jpechane commented Apr 12, 2018

ppatierno commented Apr 12, 2018

jpechane commented Apr 12, 2018

gunnarmorling commented Apr 12, 2018 •

edited

gunnarmorling commented Apr 18, 2018

jpechane commented Apr 18, 2018

gunnarmorling commented Apr 18, 2018

jpechane commented Apr 19, 2018

gunnarmorling commented Apr 19, 2018

DBZ-654 Instructions for Strimzi 0.2.0 #145

DBZ-654 Instructions for Strimzi 0.2.0 #145

Conversation

jpechane commented Apr 11, 2018

gunnarmorling left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpechane commented Apr 12, 2018

gunnarmorling commented Apr 12, 2018

ppatierno commented Apr 12, 2018

jpechane commented Apr 12, 2018

ppatierno commented Apr 12, 2018

jpechane commented Apr 12, 2018

gunnarmorling commented Apr 12, 2018 • edited

gunnarmorling commented Apr 18, 2018

jpechane commented Apr 18, 2018

gunnarmorling commented Apr 18, 2018

jpechane commented Apr 19, 2018

gunnarmorling commented Apr 19, 2018

gunnarmorling commented Apr 12, 2018 •

edited