Skip to content
This repository has been archived by the owner on Aug 19, 2024. It is now read-only.

Implemented Kafka client #685

Merged
merged 8 commits into from
Sep 24, 2020

Conversation

jeongkyun-oh
Copy link
Contributor

@jeongkyun-oh jeongkyun-oh commented Sep 23, 2020

Proposed changes

  • Implemented Kafka client
  • Publish, CreateTopic, DeleteTopic, ListTopics are implemented and used later on.

Types of changes

Please put an x in the boxes related to your change.

  • Bugfix
  • New feature or enhancement
  • Others

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the CONTRIBUTING GUIDELINES doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes ($ make test)
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules

datasync/chaindatafetcher/kafka/kafka.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka_test.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka_test.go Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka_test.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka_test.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/config.go Outdated Show resolved Hide resolved
datasync/chaindatafetcher/kafka/kafka.go Outdated Show resolved Hide resolved
@@ -347,6 +347,7 @@ func doFmt(cmdline []string) {
"--tests",
"--disable-all",
"--enable=gofmt",
"--timeout=2m",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this related to this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this, the linter is failed with timeout error.

type KafkaConfig struct {
saramaConfig *sarama.Config // kafka client configurations.
brokers []string // brokers is a list of broker URLs.
partitions int32 // partitions is the number of partitions of a topic.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make sure what I understand is correct, partition is a unit of distributed processing and replica is a unit of replication, right?

Copy link
Contributor Author

@jeongkyun-oh jeongkyun-oh Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, a topic is kinda message queue which consists of several partitions to process message in parallel. The replica(= replication factor) is how many partitions are copied to other brokers.

For example, topic1(T1) has three partitions. Broker1 is a leader for topic1-partition1(T1-P1) and the other brokers have the replicated T1-P1.

image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fully Understood :)

@jeongkyun-oh jeongkyun-oh merged commit 8159066 into klaytn:dev Sep 24, 2020
@jeongkyun-oh jeongkyun-oh deleted the KLT-135-impl-kafka-broker2 branch September 24, 2020 06:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants