Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka reporter sendBatch #148

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Conversation

sixiaobai
Copy link

Signed-off-by: wanglongfei wanglongfei@100tal.com

Signed-off-by: wanglongfei <wanglongfei@100tal.com>
@coveralls
Copy link

coveralls commented Sep 25, 2019

Coverage Status

Coverage decreased (-7.0%) to 68.456% when pulling 9907f9b on sixiaobai:master into c29478e on openzipkin:master.

@jcchavezs
Copy link
Contributor

jcchavezs commented Sep 25, 2019 via email

Signed-off-by: wanglongfei <wanglongfei@100tal.com>
const (
defaultBatchInterval = time.Second * 1 // BatchInterval in seconds
defaultBatchSize = 100
defaultMaxBacklog = 1000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeqo are these good defaults?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc we are batching 100, and max size is 1000 bytes (?).

We have these defaults in java reporter: https://github.com/openzipkin/zipkin-reporter-java/blob/3f466e56012384ee685dd5cc91012129d312289b/kafka/src/main/java/zipkin2/reporter/kafka/KafkaSender.java#L81-L90

max request size is 1MB (default in kafka producer) and but somewhere we've discuss to reduce it to have and avoid too big batches, so 500KB should be a good default.

Make sure to align producer properties to reporter ones, similar to https://github.com/openzipkin/zipkin-reporter-java/blob/3f466e56012384ee685dd5cc91012129d312289b/kafka/src/main/java/zipkin2/reporter/kafka/KafkaSender.java#L131-L140

Copy link
Author

@sixiaobai sixiaobai Oct 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeqo similar to http reporter:
When batch size reaches BatchSize, a collect will be triggered.
When batch size reaches MaxBacklog, spans from the beginning of the batch will be disposed.
So, the real batch size is dynamic, between 100 and 1000 (Exclude clock-triggered).
Assuming that the average span size is 500 bytes, so max size will be 500KB and 1000 should be a good default. The average span size is the thing we need to focus on.

Max request size is also 1MB in sarama kafka producer: https://github.com/Shopify/sarama/blob/master/config.go#L411

@@ -30,15 +32,31 @@ import (
// defaultKafkaTopic sets the standard Kafka topic our Reporter will publish
// on. The default topic for zipkin-receiver-kafka is "zipkin", see:
// https://github.com/openzipkin/zipkin/tree/master/zipkin-receiver-kafka
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: invalid url

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const (
defaultBatchInterval = time.Second * 1 // BatchInterval in seconds
defaultBatchSize = 100
defaultMaxBacklog = 1000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc we are batching 100, and max size is 1000 bytes (?).

We have these defaults in java reporter: https://github.com/openzipkin/zipkin-reporter-java/blob/3f466e56012384ee685dd5cc91012129d312289b/kafka/src/main/java/zipkin2/reporter/kafka/KafkaSender.java#L81-L90

max request size is 1MB (default in kafka producer) and but somewhere we've discuss to reduce it to have and avoid too big batches, so 500KB should be a good default.

Make sure to align producer properties to reporter ones, similar to https://github.com/openzipkin/zipkin-reporter-java/blob/3f466e56012384ee685dd5cc91012129d312289b/kafka/src/main/java/zipkin2/reporter/kafka/KafkaSender.java#L131-L140

return func(r *kafkaReporter) { r.maxBacklog = n }
}

// Topic sets the kafka topic to attach the reporter producer on.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: dup line

Signed-off-by: wanglongfei <wanglongfei@100tal.com>
@jcchavezs
Copy link
Contributor

Will check this PR with @jeqo this wednesday.

@jcchavezs
Copy link
Contributor

Should we align this number also in here @jeqo ?

@jeqo
Copy link
Member

jeqo commented Nov 11, 2019

@jcchavezs I think so. Will give a try.

@jeqo
Copy link
Member

jeqo commented Nov 11, 2019

@jcchavezs in order to apply a similar boundary as java reporter, we will need to batch based on bytes size, instead of number of elements. We can check this on a follow up PR.

Signed-off-by: wanglongfei <wanglongfei@100tal.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants