Mirror of Apache Kafka
Java Scala Python Shell Batchfile XSLT HTML
Failed to load latest commit information.
bin KAFKA-3742: (FIX) Can't run bin/connect-*.sh with -daemon flag Aug 26, 2016
checkstyle KAFKA-4008: Module "tools" should not be dependent on "core" Aug 2, 2016
clients/src KAFKA-4015; Change cleanup.policy config to accept a list of valid po… Aug 25, 2016
config KAFKA-3769: Create new sensors per-thread in KafkaStreams Aug 16, 2016
connect KAFKA-4070: implement Connect Struct.toString() Aug 26, 2016
core/src MINOR: Improve log message in `ReplicaManager.becomeLeaderOrFollower` Aug 26, 2016
docs MINOR: Update MirrorMaker docs to remove multiple --consumer.config o… Aug 26, 2016
examples KAFKA-3548: Use root locale for case transformation of constant strings Apr 21, 2016
gradle MINOR: Update rocksDB dependency to 4.9.0 Aug 25, 2016
log4j-appender/src KAFKA-3077: Enable KafkaLog4jAppender to work with SASL enabled brokers Jan 12, 2016
streams HOTFIX: disabled application-reset-tool integration test Aug 25, 2016
tests KAFKA-4016: Added join benchmarks Aug 19, 2016
tools/src/main/java/org/apache/kafka/tools KAFKA-4052: Allow passing properties file to ProducerPerformance Aug 26, 2016
vagrant MINOR: cleanup apache license in python files Jul 26, 2016
.gitignore KAFKA-3479: Add new consumer metrics documentation Aug 7, 2016
CONTRIBUTING.md KAFKA-2321; Introduce CONTRIBUTING.md Jul 27, 2015
HEADER trivial fix to add missing license header using .gradlew licenseForma… Feb 7, 2014
LICENSE KAFKA-3314: Add CDDL license to LICENSE and NOTICE file Mar 3, 2016
NOTICE KAFKA-3314: Add CDDL license to LICENSE and NOTICE file Mar 3, 2016
README.md MINOR: Add user overridden test logging events Jun 3, 2016
Vagrantfile MINOR: Add vagrant up wrapper for simple parallel bringup on aws Mar 20, 2016
build.gradle KAFKA-4082; Upgrade to Gradle 3.0 Aug 23, 2016
doap_Kafka.rdf MINOR: Remove <release> tag from doap file May 12, 2016
gradle.properties Changing version to Mar 21, 2016
gradlew KAFKA-4082; Upgrade to Gradle 3.0 Aug 23, 2016
gradlew.bat KAFKA-4082; Upgrade to Gradle 3.0 Aug 23, 2016
kafka-merge-pr.py MINOR: update new version in additional places Mar 21, 2016
release_notes.py HOTFIX: Add license information to release_notes.py Jul 28, 2016
settings.gradle KAFKA-3066: Demo Examples for Kafka Streams Jan 22, 2016
wrapper.gradle KAFKA-1490 remove gradlew initial setup output from source distributi… Sep 23, 2014


Apache Kafka

See our web site for details on the project.

You need to have Gradle and Java installed.

Kafka requires Gradle 2.0 or higher.

Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.

First bootstrap and download the wrapper

cd kafka_source_dir

Now everything else will work.

Building a jar and running it

./gradlew jar  

Follow instructions in http://kafka.apache.org/documentation.html#quickstart

Building source jar

./gradlew srcJar

Building aggregated javadoc

./gradlew aggregatedJavadoc

Building javadoc and scaladoc

./gradlew javadoc
./gradlew javadocJar # builds a javadoc jar for each module
./gradlew scaladoc
./gradlew scaladocJar # builds a scaladoc jar for each module
./gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each module

Running unit tests

./gradlew test

Forcing re-running unit tests w/o code change

./gradlew cleanTest test

Running a particular unit test

./gradlew -Dtest.single=RequestResponseSerializationTest core:test

Running a particular test method within a unit test

./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime

Running a particular unit test with log4j output

Change the log4j setting in either clients/src/test/resources/log4j.properties or core/src/test/resources/log4j.properties

./gradlew -i -Dtest.single=RequestResponseSerializationTest core:test

Generating test coverage reports

./gradlew reportCoverage

Building a binary release gzipped tar ball

./gradlew clean
./gradlew releaseTarGz

The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run:

./gradlew releaseTarGz -x signArchives

The release file can be found inside ./core/build/distributions/.

Cleaning the build

./gradlew clean

Running a task on a particular version of Scala (either 2.10.6 or 2.11.8)

Note that if building the jars with a version other than 2.10, you need to set the SCALA_BINARY_VERSION variable or change it in bin/kafka-run-class.sh to run the quick start.

You can pass either the major version (eg 2.11) or the full version (eg 2.11.8):

./gradlew -PscalaVersion=2.11 jar
./gradlew -PscalaVersion=2.11 test
./gradlew -PscalaVersion=2.11 releaseTarGz

Running a task for a specific project

This is for core, examples and clients

./gradlew core:jar
./gradlew core:test

Listing all gradle tasks

./gradlew tasks

Building IDE project

Note that this is not strictly necessary (IntelliJ IDEA has good built-in support for Gradle projects, for example).

./gradlew eclipse
./gradlew idea

Building the jar for all scala versions and for all projects

./gradlew jarAll

Running unit tests for all scala versions and for all projects

./gradlew testAll

Building a binary release gzipped tar ball for all scala versions

./gradlew releaseTarGzAll

Publishing the jar for all version of Scala and for all projects to maven

./gradlew uploadArchivesAll

Please note for this to work you should create/update ~/.gradle/gradle.properties and assign the following variables


Installing the jars to the local Maven repository

./gradlew installAll

Building the test jar

./gradlew testJar

Determining how transitive dependencies are added

./gradlew core:dependencies --configuration runtime

Determining if any dependencies could be updated

./gradlew dependencyUpdates

Running checkstyle on the java code

./gradlew checkstyleMain checkstyleTest

This will most commonly be useful for automated builds where the full resources of the host running the build and tests may not be dedicated to Kafka's build.

Common build options

The following options should be set with a -D switch, for example ./gradlew -Dorg.gradle.project.maxParallelForms=1 test.

  • org.gradle.project.mavenUrl: sets the URL of the maven deployment repository (file://path/to/repo can be used to point to a local repository).
  • org.gradle.project.maxParallelForks: limits the maximum number of processes for each task.
  • org.gradle.project.showStandardStreams: shows standard out and standard error of the test JVM(s) on the console.
  • org.gradle.project.skipSigning: skips signing of artifacts.
  • org.gradle.project.testLoggingEvents: unit test events to be logged, separated by comma. For example ./gradlew -Dorg.gradle.project.testLoggingEvents=started,passed,skipped,failed test

Running in Vagrant

See vagrant/README.md.


Apache Kafka is interested in building the community; we would welcome any thoughts or patches. You can reach us on the Apache mailing lists.

To contribute follow the instructions here: