Skip to content
Mirror of Apache Kafka
Java Scala Python Shell Batchfile XSLT HTML
Latest commit 1e4dd66 Apr 30, 2016 @dpkp dpkp committed with ewencp HOTFIX: Fix main classpath libs glob for release (fixup KAFKA-3615 re…
…gression)

bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2.

Author: Dana Powers <dana.powers@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1302 from dpkp/KAFKA-3615-fix
Failed to load latest commit information.
bin HOTFIX: Fix main classpath libs glob for release (fixup KAFKA-3615 re… Apr 30, 2016
checkstyle KAFKA-3611: Remove warnings when using reflections Apr 28, 2016
clients/src KAFKA-3644; Use Boolean protocol type for StopReplicaRequest delete_p… Apr 29, 2016
config KAFKA-3582; Remove references to Copcyat from Kafka Connect property … Apr 18, 2016
connect KAFKA-3459: Returning zero task configurations from a connector does … Apr 29, 2016
core/src KAFKA-3618; Handle ApiVersionsRequest before SASL authentication Apr 29, 2016
docs MINOR: Typo fixes in ReplicaFetchMaxBytesDoc Apr 26, 2016
examples KAFKA-3548: Use root locale for case transformation of constant strings Apr 20, 2016
gradle KAFKA-3557; Update rocksdb to 4.4.1 and patch updates to snappy and s… Apr 14, 2016
log4j-appender/src KAFKA-3077: Enable KafkaLog4jAppender to work with SASL enabled brokers Jan 11, 2016
streams HOTFIX: Fix equality semantics of KeyValue Apr 29, 2016
tests KAFKA-3597; Query ConsoleConsumer and VerifiableProducer if they shut… Apr 29, 2016
tools/src/main/java/org/apache/kafka/tools KAFKA-3597; Query ConsoleConsumer and VerifiableProducer if they shut… Apr 29, 2016
vagrant KAFKA-3490; Multiple version support for ducktape performance tests Apr 13, 2016
.gitignore KAFKA-3361: Initial protocol documentation page and generation Mar 9, 2016
CONTRIBUTING.md KAFKA-2321; Introduce CONTRIBUTING.md Jul 27, 2015
HEADER trivial fix to add missing license header using .gradlew licenseForma… Feb 7, 2014
LICENSE KAFKA-3314: Add CDDL license to LICENSE and NOTICE file Mar 3, 2016
NOTICE KAFKA-3314: Add CDDL license to LICENSE and NOTICE file Mar 3, 2016
README.md KAFKA-3451: Add basic HTML coverage report generation to gradle Mar 31, 2016
Vagrantfile MINOR: Add vagrant up wrapper for simple parallel bringup on aws Mar 20, 2016
build.gradle KAFKA-3612: Added structure for integration tests Apr 27, 2016
doap_Kafka.rdf trivial change to add kafka doap project file Apr 11, 2014
gradle.properties Changing version to 0.10.1.0-SNAPSHOT Mar 21, 2016
gradlew MINOR: Upgrade to Gradle 2.13 Apr 26, 2016
gradlew.bat MINOR: Upgrade to Gradle 2.13 Apr 26, 2016
kafka-merge-pr.py MINOR: update new version in additional places Mar 21, 2016
settings.gradle KAFKA-3066: Demo Examples for Kafka Streams Jan 22, 2016
wrapper.gradle KAFKA-1490 remove gradlew initial setup output from source distributi… Sep 23, 2014

README.md

Apache Kafka

See our web site for details on the project.

You need to have Gradle and Java installed.

Kafka requires Gradle 2.0 or higher.

Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.

First bootstrap and download the wrapper

cd kafka_source_dir
gradle

Now everything else will work.

Building a jar and running it

./gradlew jar  

Follow instructions in http://kafka.apache.org/documentation.html#quickstart

Building source jar

./gradlew srcJar

Building javadocs and scaladocs

./gradlew javadoc
./gradlew javadocJar # builds a jar from the javadocs
./gradlew scaladoc
./gradlew scaladocJar # builds a jar from the scaladocs
./gradlew docsJar # builds both javadoc and scaladoc jar

Running unit tests

./gradlew test

Forcing re-running unit tests w/o code change

./gradlew cleanTest test

Running a particular unit test

./gradlew -Dtest.single=RequestResponseSerializationTest core:test

Running a particular test method within a unit test

./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime

Running a particular unit test with log4j output

Change the log4j setting in either clients/src/test/resources/log4j.properties or core/src/test/resources/log4j.properties

./gradlew -i -Dtest.single=RequestResponseSerializationTest core:test

Generating test coverage reports

./gradlew reportCoverage

Building a binary release gzipped tar ball

./gradlew clean
./gradlew releaseTarGz

The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run:

./gradlew releaseTarGz -x signArchives

The release file can be found inside ./core/build/distributions/.

Cleaning the build

./gradlew clean

Running a task on a particular version of Scala (either 2.10.6 or 2.11.8)

Note that if building the jars with a version other than 2.10, you need to set the SCALA_BINARY_VERSION variable or change it in bin/kafka-run-class.sh to run the quick start.

You can pass either the major version (eg 2.11) or the full version (eg 2.11.8):

./gradlew -PscalaVersion=2.11 jar
./gradlew -PscalaVersion=2.11 test
./gradlew -PscalaVersion=2.11 releaseTarGz

Running a task for a specific project

This is for core, examples and clients

./gradlew core:jar
./gradlew core:test

Listing all gradle tasks

./gradlew tasks

Building IDE project

Note that this is not strictly necessary (IntelliJ IDEA has good built-in support for Gradle projects, for example).

./gradlew eclipse
./gradlew idea

Building the jar for all scala versions and for all projects

./gradlew jarAll

Running unit tests for all scala versions and for all projects

./gradlew testAll

Building a binary release gzipped tar ball for all scala versions

./gradlew releaseTarGzAll

Publishing the jar for all version of Scala and for all projects to maven

./gradlew uploadArchivesAll

Please note for this to work you should create/update ~/.gradle/gradle.properties and assign the following variables

mavenUrl=
mavenUsername=
mavenPassword=
signing.keyId=
signing.password=
signing.secretKeyRingFile=

Installing the jars to the local Maven repository

./gradlew installAll

Building the test jar

./gradlew testJar

Determining how transitive dependencies are added

./gradlew core:dependencies --configuration runtime

Determining if any dependencies could be updated

./gradlew dependencyUpdates

Running checkstyle on the java code

./gradlew checkstyleMain checkstyleTest

This will most commonly be useful for automated builds where the full resources of the host running the build and tests may not be dedicated to Kafka's build.

Common build options

The following options should be set with a -D switch, for example ./gradlew -Dorg.gradle.project.maxParallelForms=1 test.

  • org.gradle.project.mavenUrl: sets the URL of the maven deployment repository (file://path/to/repo can be used to point to a local repository).
  • org.gradle.project.maxParallelForks: limits the maximum number of processes for each task.
  • org.gradle.project.showStandardStreams: shows standard out and standard error of the test JVM(s) on the console.
  • org.gradle.project.skipSigning: skips signing of artifacts.

Running in Vagrant

See vagrant/README.md.

Contribution

Apache Kafka is interested in building the community; we would welcome any thoughts or patches. You can reach us on the Apache mailing lists.

To contribute follow the instructions here:

Something went wrong with that request. Please try again.