Skip to content

Commit fb79d49

Browse files
stefans-elastickhushijain21Linu-Elias
authored
update kafka version used in testing to 3.6.0 (#42286)
* update kafka version used in testing * some environment changes * boostrap * 3.6.0 versio change * Update metricbeat/module/kafka/_meta/healthcheck.sh Co-authored-by: Linu-Elias <linu.elias@elastic.co> * uncomment compose waiting * kafka 3.6.0 fix * fix mistake in path to adminclient.properties * fix kafka image link in docker-compose * fix libbeats kafka docker testing image * bump kafka to 2.13 * update kafka version in partition.go * update kafka version in docs --------- Co-authored-by: Khushi Jain <khushi.jain@elastic.co> Co-authored-by: Linu-Elias <linu.elias@elastic.co>
1 parent c156390 commit fb79d49

File tree

20 files changed

+42
-38
lines changed

20 files changed

+42
-38
lines changed

‎libbeat/docker-compose.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
version: '2.3'
21
services:
32
# This is a proxy used to block "docker-compose up" until all services are healthy.
43
# See: https://github.com/docker/compose/issues/4369

‎metricbeat/docker-compose.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
version: '2.3'
2-
31
services:
42
beat:
53
build: ${PWD}/.

‎metricbeat/docs/modules/kafka.asciidoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allo
3535
[float]
3636
=== Compatibility
3737

38-
This module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, and 2.2.2.
38+
This module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, 2.2.2 and 3.6.0.
3939

4040
The Broker, Producer, Consumer metricsets require <<metricbeat-module-jolokia,Jolokia>> to fetch JMX metrics. Refer to the link for Jolokia's compatibility notes.
4141

‎metricbeat/module/kafka/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,10 @@ To bring this container up simply run the tests for Kafka module:
2020
After the tests have been completed, the Kafka container should be still running. Verify with:
2121

2222
```console
23-
707b50334835 docker.elastic.co/integrations-ci/beats-kafka:2.2.2-2 "/run.sh" 2 minutes ago Up 2 minutes (healthy) 2181/tcp, 0.0.0.0:32785->8774/tcp, 0.0.0.0:32784->8775/tcp, 0.0.0.0:32783->8779/tcp, 0.0.0.0:32782->9092/tcp kafka_a035cf4c6889705a_kafka_1
23+
707b50334835 docker.elastic.co/integrations-ci/beats-kafka:3.6.0-2 "/run.sh" 2 minutes ago Up 2 minutes (healthy) 2181/tcp, 0.0.0.0:32785->8774/tcp, 0.0.0.0:32784->8775/tcp, 0.0.0.0:32783->8779/tcp, 0.0.0.0:32782->9092/tcp kafka_a035cf4c6889705a_kafka_1
2424
```
2525

26-
In order to identify to which port the Broker is listening on one should check in the logs of the container and find
26+
In order to identify to which port the Broker is listening on one should check in the logs of the container and find
2727
the advertised address:
2828

2929
```console
@@ -39,7 +39,7 @@ listeners = INSIDE://localhost:9091,OUTSIDE://0.0.0.0:9092
3939
```
4040

4141
So here in this example the host we should in the module's config is `localhost:32778`.
42-
Note that this is different between MAC and Linux machines. The above is the case for the MAC machine, and here is how
42+
Note that this is different between MAC and Linux machines. The above is the case for the MAC machine, and here is how
4343
the respective address for a LINUX machine should look like:
4444

4545
```console
@@ -60,7 +60,7 @@ This was needed before moving the metricbeat docker used in CI to host network,
6060

6161
#### Configuring Kafka module
6262
In order to configure the Module we will use the advertised addressed to connect to the broker and the credentials
63-
that are also used for the tests
63+
that are also used for the tests
6464
(see [test config](https://github.com/elastic/beats/blob/6c279ebf2789655725889f37820c959a8f2ea969/metricbeat/module/kafka/consumergroup/consumergroup_integration_test.go#L39)).
6565
Here is how the config should look like (in a MAC):
6666

‎metricbeat/module/kafka/_meta/Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,13 @@ RUN apt-get update && apt-get install -y curl openjdk-8-jre-headless netcat-open
1818
RUN mkdir -p ${KAFKA_LOGS_DIR} && mkdir -p ${KAFKA_HOME} && \
1919
curl -J -L -s -f -o - https://github.com/kadwanev/retry/releases/download/1.0.1/retry-1.0.1.tar.gz | tar xfz - -C /usr/local/bin && \
2020
retry --min 1 --max 180 -- curl -J -L -s -f --show-error -o $INSTALL_DIR/kafka.tgz \
21-
"https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_2.11-${KAFKA_VERSION}.tgz" && \
21+
"https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_2.13-${KAFKA_VERSION}.tgz" && \
2222
tar xzf ${INSTALL_DIR}/kafka.tgz -C ${KAFKA_HOME} --strip-components 1
2323

2424
RUN retry --min 1 --max 180 -- curl -J -L -s -f --show-error -o /opt/jolokia-jvm-1.5.0-agent.jar \
2525
http://search.maven.org/remotecontent\?filepath\=org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar
2626

27+
ADD adminclient.properties /kafka/bin/adminclient.properties
2728
ADD kafka_server_jaas.conf /etc/kafka/server_jaas.conf
2829
ADD jaas-kafka-client-producer.conf /kafka/bin/jaas-kafka-client-producer.conf
2930
ADD sasl-producer.properties /kafka/bin/sasl-producer.properties

‎metricbeat/module/kafka/_meta/adminclient.properties‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
security.protocol=SASL_PLAINTEXT
2+
sasl.mechanism=PLAIN
3+
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=admin password=admin-secret;

‎metricbeat/module/kafka/_meta/docs.asciidoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allo
2424
[float]
2525
=== Compatibility
2626

27-
This module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, and 2.2.2.
27+
This module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, 2.2.2 and 3.6.0.
2828

2929
The Broker, Producer, Consumer metricsets require <<metricbeat-module-jolokia,Jolokia>> to fetch JMX metrics. Refer to the link for Jolokia's compatibility notes.
3030

‎metricbeat/module/kafka/_meta/healthcheck.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@
44

55
TOPIC="foo-`date '+%s-%N'`"
66

7-
${KAFKA_HOME}/bin/kafka-topics.sh --zookeeper=127.0.0.1:2181 --create --partitions 1 --topic "${TOPIC}" --replication-factor 1
7+
${KAFKA_HOME}/bin/kafka-topics.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --create --partitions 1 --topic "${TOPIC}" --replication-factor 1
88
rc=$?
99
if [[ $rc != 0 ]]; then
1010
exit $rc
1111
fi
1212

13-
${KAFKA_HOME}/bin/kafka-topics.sh --zookeeper=127.0.0.1:2181 --delete --topic "${TOPIC}"
13+
${KAFKA_HOME}/bin/kafka-topics.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --delete --topic "${TOPIC}"
1414
exit 0

‎metricbeat/module/kafka/_meta/run.sh‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ wait_for_port() {
3434
nc -z localhost $port
3535
}
3636

37-
${KAFKA_HOME}/bin/kafka-topics.sh --zookeeper=127.0.0.1:2181 --create --partitions 1 --topic test --replication-factor 1
37+
${KAFKA_HOME}/bin/kafka-topics.sh --bootstrap-server localhost:9091 --create --partitions 1 --topic test --replication-factor 1 --command-config ${KAFKA_HOME}/bin/adminclient.properties
3838

3939
echo "Starting ZooKeeper"
4040
${KAFKA_HOME}/bin/zookeeper-server-start.sh ${KAFKA_HOME}/config/zookeeper.properties &
@@ -44,7 +44,7 @@ echo "Starting Kafka broker"
4444
mkdir -p ${KAFKA_LOGS_DIR}
4545
export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/server_jaas.conf -javaagent:/opt/jolokia-jvm-1.5.0-agent.jar=port=8779,host=0.0.0.0"
4646
${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server.properties \
47-
--override authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer \
47+
--override authorizer.class.name=kafka.security.authorizer.AclAuthorizer \
4848
--override super.users=User:admin \
4949
--override sasl.enabled.mechanisms=PLAIN \
5050
--override sasl.mechanism.inter.broker.protocol=PLAIN \
@@ -53,6 +53,7 @@ ${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server.properties \
5353
--override advertised.listeners=INSIDE://localhost:9091,OUTSIDE://$KAFKA_ADVERTISED_HOST \
5454
--override listener.security.protocol.map=INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT \
5555
--override inter.broker.listener.name=INSIDE \
56+
--override zookeeper.set.acl=false \
5657
--override logs.dir=${KAFKA_LOGS_DIR} &
5758

5859
wait_for_port 9092
@@ -62,22 +63,23 @@ wait_for_port 8779
6263
echo "Kafka load status code $?"
6364

6465
# ACLS used to prepare tests
65-
${KAFKA_HOME}/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:producer --operation All --cluster --topic '*' --group '*'
66-
${KAFKA_HOME}/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:consumer --operation All --cluster --topic '*' --group '*'
66+
${KAFKA_HOME}/bin/kafka-acls.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --add --allow-principal User:producer --operation All --cluster --topic '*' --group '*'
67+
${KAFKA_HOME}/bin/kafka-acls.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --add --allow-principal User:consumer --operation All --cluster --topic '*' --group '*'
6768

6869
# Minimal ACLs required by metricbeat. If this needs to be changed, please update docs too
69-
${KAFKA_HOME}/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Describe --group '*'
70-
${KAFKA_HOME}/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Read --topic '*'
70+
${KAFKA_HOME}/bin/kafka-acls.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --add --allow-principal User:stats --operation Describe --group '*'
71+
${KAFKA_HOME}/bin/kafka-acls.sh --bootstrap-server localhost:9091 --command-config ${KAFKA_HOME}/bin/adminclient.properties --add --allow-principal User:stats --operation Read --topic '*'
7172

7273
touch /tmp/.acls_loaded
7374

74-
75+
echo "Kafka producer start"
7576
# Start a forever producer
7677
{ while sleep 1; do echo message; done } | KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/bin/jaas-kafka-client-producer.conf -javaagent:/opt/jolokia-jvm-1.5.0-agent.jar=port=8775,host=0.0.0.0" \
7778
${KAFKA_HOME}/bin/kafka-console-producer.sh --topic test --broker-list localhost:9091 --producer.config ${KAFKA_HOME}/bin/sasl-producer.properties > /dev/null &
7879

7980
wait_for_port 8775
8081

82+
echo "Kafka consumer start"
8183
# Start a forever consumer
8284
KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/bin/jaas-kafka-client-consumer.conf -javaagent:/opt/jolokia-jvm-1.5.0-agent.jar=port=8774,host=0.0.0.0" \
8385
${KAFKA_HOME}/bin/kafka-console-consumer.sh --topic=test --bootstrap-server=localhost:9091 --consumer.config ${KAFKA_HOME}/bin/sasl-producer.properties > /dev/null &

‎metricbeat/module/kafka/_meta/supported-versions.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
variants:
2+
- KAFKA_VERSION: 3.6.0
23
- KAFKA_VERSION: 2.2.2
34
- KAFKA_VERSION: 2.1.1
45
- KAFKA_VERSION: 2.0.0

0 commit comments

Comments
 (0)