Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata snapshot consumer #2

Merged
merged 3 commits into from
Aug 20, 2014
Merged

Metadata snapshot consumer #2

merged 3 commits into from
Aug 20, 2014

Conversation

krishnap
Copy link

  • A simple consumer to consume metadata from kafka and write our the difference
  • in the offsets of the partitions from previous run. This is used for
  • accounting purposes to figure out how many events are expected to be read
  • from kafka between successive runs of this consumer.

@brndnmtthws @andykram @wmoss @mrdmnd @jamesmay

*/

class OffsetInfo implements WritableComparable<OffsetInfo> {
private String topic;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eight space tabs? Can we go smaller?

@krishnap
Copy link
Author

I reformatted after moving to intelliJ, and it addressed the formatting comments you guys had.

Other comments:
** fixed the typos
** the default magic numbers are what we currently set in our camus.properties file. so i picked those values. If not these, i still need to provide some default for these parameters.
** command line parsing + logging is added

thanks for the comments.

oneJsonNode.put("difference", diff);
jsonData.add(oneJsonNode);

System.out.println(String.format("Topic: %s OffsetDifference: %d", pk, diff));

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use log here.

@brndnmtthws
Copy link

Maybe squash all commits into one?

Otherwise, looks better.

@krishnap
Copy link
Author

I'd like to schedule this consumer to run today, so i'm going to merge this PR. Feel free to send me more comments.

krishnap added a commit that referenced this pull request Aug 20, 2014
@krishnap krishnap merged commit 8aba179 into master Aug 20, 2014
@krishnap krishnap deleted the metadata-fetcher branch August 20, 2014 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants