Skip to content
This repository has been archived by the owner. It is now read-only.

add Offset() to PartitionConsumer interface #221

Merged
merged 3 commits into from Mar 16, 2018
Merged

Conversation

imjustfly
Copy link
Contributor

@imjustfly imjustfly commented Feb 26, 2018

Sometimes we need to know the initial offset of a PartitionConsumer

Copy link
Member

@dim dim left a comment

Hey, thanks for this, could you please fix the naming? Thanks again!

partitions.go Outdated

// Offset returns the offset used for creating the PartitionConsumer instance.
// Offset can be a literal offset, or OffsetNewest, or OffsetOldest
Offset() int64
Copy link
Member

@dim dim Mar 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please rename this method to InitialOffset?

Copy link
Contributor Author

@imjustfly imjustfly Mar 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I will do it.

@imjustfly
Copy link
Contributor Author

imjustfly commented Mar 8, 2018

OK. I fixed the naming.

dim
dim approved these changes Mar 16, 2018
@dim dim merged commit d985926 into bsm:master Mar 16, 2018
1 check passed
Praveen-Upadhyay added a commit to ContinuumLLC/godep-bsm-sarama-cluster that referenced this issue May 9, 2018
* Return error instead of returning nil

* Make Close idempotent

* CI update

* Auto-discover topics

* fix test due to sarama's change in sync_producer's config

* fix the doc for Consumer.Notifications

* Bump deps

* Add HighWaterMarks support

ref: HighWaterMarks returns the current high water marks for each topic and partition
https://godoc.org/github.com/Shopify/sarama#Consumer

* Bump Sarama dependency, use latest Kafka version

* Allow to stash offsets when processing batches

* Added support method

* Subscribe to new topics

* Add troubleshooting information regarding consumer not receiving messages

* Allow to subscribe to topics via regular expressions

* Formatting

* Fix troubleshooting formatting.

* Added example

* Handle authz limited to specific topics

* express it a bit simpler

* move metadata refresh to consumer method

* Fix comparison operator

* Allow custom group member UserData to be included when joining the group

* Commit every time, even if not dirty.
Updated README and CI config.

* Updated example

* Fix off by one error when marking offset

* Updated README

* Test with Kafka 0.9

* Partially revert bsm#117. Force-commit when Consumer.Offsets.Retention is set

* Remove retention time workaround. Offset retention must be > than log retention.

* Allow sarama-cluster offset synchronization dwell time to be configured separately from Sarama MaxProcessingTime

* Fix typo and dwell time validation error handling

* Remove sleep and replace with select statement. Switch from dwell time as pointer to value.

* Switch sleep to select with speedy bailout on close

* Better config validation

* Ported from glide to dep

* Fix CI

* Fix CI

* Ensure notifications are issued (and consumed) before rebalancing is completed

* Fix CI

* Bump patch version#

* Simpler tests

* Fix seeding

* Test with Go 1.9

* Remove delay

* Reduce memory requirements

* Add consumer test

* Force rebalance when partition consumers exit with an error

* Adding notifications on rebalance start

* Stop sharing clients as this is not something Kafka allows us to do

* Expose individual partitions

* Respond to feedback

* Adding whitelist example

* Addressing feedback. Always multiplex errors

* Updated README

* Allow to re-use clients

* Adding comment to clarify usage

* Update CI settings

* Make Close thread-safe

* More robust

* Fix race, added clarifying comment to CommitOffsets

* Fix deps

* Fix the kafka source - 0.9.0.1 testing broken

* Support Kafka 1.0

* Drain channels on Close

* Allow marking of earlier offsets

This commit is in alignment with the sarama library commits
Shopify/sarama@b966238#diff-3ca6d659defd100fe2de43adf2b8f41e
Shopify/sarama@96fa1c8#diff-3ca6d659defd100fe2de43adf2b8f41e

* Address comments and add tests

* Ensure we have random topic name and reduce time

* Fix test timing for kafka ≥ 0.11.0

* Remove unncessary expect clause

* Address comments + gofmt

* Unify PartitionConsumer interface

* New release

* Fix NPE on ResetOffsets and MarkOffsets

* Use correct request proto version

* Expose offset methods on partition consumers

* Fix/remove test

* Bump CI config

* Fewer builds

* add Offset() to PartitionConsumer interface (bsm#221)

* add Offset() to PartitionConsumer interface

* rename Offset to InitialOffset
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants