Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' of github.com:linkedin-sna/sna-page
- Loading branch information
Showing
33 changed files
with
434 additions
and
1,356 deletions.
There are no files selected for viewing
Binary file not shown.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,4 @@ | ||
| <?php require "includes/project_info.php" ?> | ||
| <?php require "../includes/header.php" ?> | ||
| <?php include "../includes/advert.php" ?> | ||
|
|
||
| <h2> Configuration </h2> | ||
|
|
||
| <h3> Important configuration properties for Kafka broker: </h3> | ||
|
|
||
| <p>More details about server configuration can be found in the scala class <code>kafka.server.KafkaConfig</code>.</p> | ||
|
|
||
| <?php include('includes/server_config.php'); ?> | ||
|
|
||
| <h3> Important configuration properties for the high-level consumer: </h3> | ||
|
|
||
| <p>More details about consumer configuration can be found in the scala class <code>kafka.consumer.ConsumerConfig</code>.</p> | ||
|
|
||
| <?php include('includes/consumer_config.php'); ?> | ||
|
|
||
| <h3> Important configuration properties for the producer: </h3> | ||
|
|
||
| <p>More details about producer configuration can be found in the scala class <code>kafka.producer.ProducerConfig</code>.</p> | ||
|
|
||
| <?php include('includes/producer_config.php'); ?> | ||
|
|
||
| <?php require "../includes/footer.php" ?> | ||
| <?php | ||
| header( "HTTP/1.1 301 Moved Permanently" ); | ||
| header('Location: http://incubator.apache.org/kafka/configuration.html'); | ||
| ?> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| <?php | ||
| header( "HTTP/1.1 301 Moved Permanently" ); | ||
| header('Location: http://incubator.apache.org/kafka/contact.html'); | ||
| ?> |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| <?php require "includes/project_info.php" ?> | ||
| <?php require "../includes/header.php" ?> | ||
| <?php include "../includes/advert.php" ?> | ||
|
|
||
| <h2>Downloads</h2> | ||
| <ul> | ||
| <li><a href="downloads/kafka-0.05.zip">kafka-0.05.zip</a></li> | ||
| <li><a href="downloads/kafka-0.6.RC1.zip">kafka-0.6.RC1.zip</a></li> | ||
| <li><a href="downloads/kafka-0.6.RC2.zip">kafka-0.6.RC2.zip</a></li> | ||
| <li><a href="downloads/kafka-0.6.zip">kafka-0.6.zip</a></li> | ||
| </ul> | ||
|
|
||
| <h2>Release Notes</h2> | ||
| <ul> | ||
| <li><a href="downloads/v0.6-RC1-release-notes.html">v0.6.RC1 release notes</a></li> | ||
| <li><a href="downloads/v0.6-RC2-release-notes.html">v0.6.RC2 release notes</a></li> | ||
| <li><a href="downloads/v0.6-release-notes.html">v0.6 release notes</a></li> | ||
| </ul> | ||
|
|
||
| <h2>API docs</h2> | ||
| <ul> | ||
| <li><a href="downloads/0.6-api/index.html">0.6 API docs</a></li> | ||
| </ul> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,4 @@ | ||
| <?php require "includes/project_info.php" ?> | ||
| <?php require "../includes/header.php" ?> | ||
| <?php include "../includes/advert.php" ?> | ||
|
|
||
| <h2>Frequently asked questions</h3> | ||
| <ol> | ||
| <li> <h3> Why does my consumer get InvalidMessageSizeException? </h3> | ||
| This typically means that the "fetch size" of the consumer is too small. Each time the consumer pulls data from the broker, it reads bytes up to a configured limit. If that limit is smaller than the largest single message stored in Kafka, the consumer can't decode the message properly and will throw an InvalidMessageSizeException. To fix this, increase the limit by setting the property "fetch.size" properly in config/consumer.properties. The default fetch.size is 300,000 bytes. | ||
|
|
||
| <li> <h3> On EC2, why can't my high-level consumers connect to the brokers? </h3> | ||
| When a broker starts up, it registers its host ip in ZK. The high-level consumer later uses the registered host ip to establish the socket connection to the broker. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, in EC2, the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the "hostname" property in server.properties. | ||
|
|
||
| </ol> | ||
| <?php | ||
| header( "HTTP/1.1 301 Moved Permanently" ); | ||
| header('Location: http://incubator.apache.org/kafka/faq.html'); | ||
| ?> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,4 @@ | ||
| <?php require "includes/project_info.php" ?> | ||
| <?php require "../includes/header.php" ?> | ||
| <?php include "../includes/advert.php" ?> | ||
|
|
||
| <h2>Kafka is a distributed publish/subscribe messaging system</h2> | ||
| <p> | ||
| Kafka is a distributed publish-subscribe messaging system. It is designed to support the following | ||
| <ul> | ||
| <li>Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.</li> | ||
| <li>High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.</li> | ||
| <li>Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.</li> | ||
| <li>Support for parallel data load into Hadoop.</li> | ||
| </ul> | ||
|
|
||
| Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines. | ||
| </p> | ||
|
|
||
| <p> | ||
| The use for activity stream processing makes Kafka comparable to <a href="https://github.com/facebook/scribe">Facebook's Scribe</a> or <a href="https://github.com/cloudera/flume">Cloudera's Flume</a>, though the architecture and primitives are very different for these systems and make Kafka more comparable to a traditional messaging system. See our <a href="design.php">design</a> page for more details. | ||
| </p> | ||
|
|
||
| <p> | ||
| This is a new project, and we are interested in building the community; we would welcome any thoughts or patches. You can reach us <a href="http://groups.google.com/group/kafka-dev">here<a/>. | ||
|
|
||
| <?php require "../includes/footer.php" ?> | ||
| <?php | ||
| header( "HTTP/1.1 301 Moved Permanently" ); | ||
| header('Location: http://incubator.apache.org/kafka/index.html'); | ||
| ?> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,87 +1,4 @@ | ||
| <?php require "includes/project_info.php" ?> | ||
| <?php require "../includes/header.php" ?> | ||
| <?php include "../includes/advert.php" ?> | ||
|
|
||
| <h2>Performance Results</h2> | ||
| <p>The following tests give some basic information on Kafka throughput as the number of topics, consumers and producers and overall data size varies. Since Kafka nodes are independent, these tests are run with a single producer, consumer, and broker machine. Results can be extrapolated for a larger cluster. | ||
| </p> | ||
|
|
||
| <p> | ||
| We run producer and consumer tests separately to isolate their performance. For the consumer these tests test <i>cold</i> performance, that is consuming a large uncached backlog of messages. Simultaneous production and consumption tends to help performance since the cache is hot. | ||
| </p> | ||
|
|
||
| <p>We took below setting for some of the parameters:</p> | ||
|
|
||
| <ul> | ||
| <li>message size = 200 bytes</li> | ||
| <li>batch size = 200 messages</li> | ||
| <li>fetch size = 1MB</li> | ||
| <li>flush interval = 600 messages</li> | ||
| </ul> | ||
|
|
||
| In our performance tests, we run experiments to answer below questions. | ||
| <h3>What is the producer throughput as a function of batch size?</h3> | ||
| <p>We can push about 50MB/sec to the system. However, this number changes with the batch size. The below graphs show the relation between these two quantities.<p> | ||
| <p><span style="" class="image-wrap"><img border="0" src="images/onlyBatchSize.jpg" width="500" height="300"/></span><br /></p> | ||
|
|
||
| <h3>What is the consumer throughput?</h3> | ||
| <p>According to our experiments, we can consume about 100M/sec from a broker and the total does not seem to change much as we increase | ||
| the number of consumer threads.<p> | ||
| <p><span style="" class="image-wrap"><img border="0" src="images/onlyConsumer.jpg" width="500" height="300"/></span> </p> | ||
|
|
||
| <h3> Does data size effect our performance? </h3> | ||
| <p><span style="" class="image-wrap"><img border="0" src="images/dataSize.jpg" width="500" height="300"/></span><br /></p> | ||
|
|
||
| <h3>What is the effect of the number of producer threads on producer throughput?</h3> | ||
| <p>We are able to max out production with only a few threads.<p> | ||
| <p><span style="" class="image-wrap"><img border="0" src="images/onlyProducer.jpg" width="500" height="300"/></span><br /></p> | ||
|
|
||
| <h3> What is the effect of number of topics on producer throughput?</h3> | ||
| <p>Based on our experiments, the number of topic has a minimal effect on the total data produced. | ||
| The below graph is an experiment where we used 40 producers and varied the number of topics<p> | ||
|
|
||
| <p><span style="" class="image-wrap"><img border="0" src="images/onlyTopic.jpg" width="500" height="300"/></span><br /></p> | ||
|
|
||
| <h2>How to Run a Performance Test</h2> | ||
|
|
||
| <p>The performance related code is under perf folder. To run the simulator :</p> | ||
|
|
||
| <p> ../run-simulator.sh -kafkaServer=localhost -numTopic=10 -reportFile=report-html/data -time=15 -numConsumer=20 -numProducer=40 -xaxis=numTopic</p> | ||
|
|
||
| <p>It will run a simulator with 40 producer and 20 consumer threads | ||
| producing/consuming from a local kafkaserver. The simulator is going to | ||
| run 15 minutes and the results are going to be saved under | ||
| report-html/data</p> | ||
|
|
||
| <p>and they will be plotted from there. Basically it will write MB of | ||
| data consumed/produced, number of messages consumed/produced given a | ||
| number of topic and report.html will plot the charts.</p> | ||
|
|
||
|
|
||
| <p>Other parameters include numParts, fetchSize, messageSize.</p> | ||
|
|
||
| <p>In order to test how the number of topic affects the performance the below script can be used (it is under utl-bin)</p> | ||
|
|
||
|
|
||
|
|
||
| <p>#!/bin/bash<br /> | ||
|
|
||
| for i in 1 10 20 30 40 50;<br /> | ||
|
|
||
| do<br /> | ||
|
|
||
| ../kafka-server.sh server.properties 2>&1 >kafka.out&<br /> | ||
| sleep 60<br /> | ||
| ../run-simulator.sh -kafkaServer=localhost -numTopic=$i -reportFile=report-html/data -time=15 -numConsumer=20 -numProducer=40 -xaxis=numTopic<br /> | ||
| ../stop-server.sh<br /> | ||
| rm -rf /tmp/kafka-logs<br /> | ||
|
|
||
| sleep 300<br /> | ||
|
|
||
| done</p> | ||
|
|
||
|
|
||
|
|
||
| <p>The charts similar to above graphs can be plotted with report.html automatically.</p> | ||
|
|
||
| <?php require "../includes/footer.php" ?> | ||
| <?php | ||
| header( "HTTP/1.1 301 Moved Permanently" ); | ||
| header('Location: http://incubator.apache.org/kafka/performance.html'); | ||
| ?> |
Oops, something went wrong.