How to run Apache Kafka using IntelliJ IDEA

[aCoder2013]

本文简单记录一下如何搭建Apache Kafka 开发环境

前言

我这里用的是MBP，如果是Windows用户那么下面的命令以及一些路径可能会有差异，比如脚本后缀应该是.bat.

安装相关库&&工具

1. JDK1.8
2. Gradle
3. Zookeeper

构建Kafka

1. https://github.com/apache/kafka， 下载源码
2. 执行以下命令

cd kafka
gradle idea 
#这里会下载一坨依赖，慢慢等，看到Build Successful说明成功了

引入IDE

1. 首先需要安装Scala插件
   

2. 配置日志，这里需要将根目录下的conf/log4j.properties拷贝到/core/src/main/resources/log4j.properties,resources文件夹不存在则新建一个

3. 配置运行时设置,直接参考下图即可
   

4. 点击运行，第一次启动时Idea会先编译整个项目，因此可能会比较慢，日志如下则表示启动成功:

[2017-11-15 21:11:47,436] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2017-11-15 21:11:47,436] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-11-15 21:11:47,470] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2017-11-15 21:11:47,504] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-11-15 21:11:47,509] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-11-15 21:11:47,510] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(192.168.1.103,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-11-15 21:11:47,515] WARN Error while loading kafka-version.properties :null (org.apache.kafka.common.utils.AppInfoParser)
[2017-11-15 21:11:47,516] INFO Kafka version : unknown (org.apache.kafka.common.utils.AppInfoParser)
[2017-11-15 21:11:47,516] INFO Kafka commitId : unknown (org.apache.kafka.common.utils.AppInfoParser)
[2017-11-15 21:11:47,517] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

验证

1. 首先来创建一个topic,单个replication，单个partitions

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Created topic "test".

现在就可以看到topic已经创建成功了:

bin/kafka-topics.sh --list --zookeeper localhost:2181                                                                                                                                                                                                                !10280
test

2. 发几个消息试试

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Hello World
你好,世界！

3. 新开一个窗口，用于消费消息:

> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Hello World
你好,世界！