Skip to content
This repository has been archived by the owner on Oct 30, 2020. It is now read-only.

reset pKey when message#key is null #167

Merged
merged 2 commits into from Feb 21, 2015
Merged

reset pKey when message#key is null #167

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f48aacc
reset pKey when message#key is null
jimjh Jan 27, 2015
3695ea5
add logging when message#key is null
Feb 21, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions camus-etl-kafka/src/main/java/com/linkedin/camus/etl/kafka/common/KafkaReader.java
View file
Open in desktop
Expand Up @@ -116,6 +116,9 @@ public boolean getNext(EtlKey key, BytesWritable payload, BytesWritable pKey) th
bytes = new byte[origSize];
buf.get(bytes, buf.position(), origSize);
pKey.set(bytes, 0, origSize);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually when do you see message.key() is null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It happened to us in production.

In Message.scala, one of the constructors sets key to null.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a logging when the key is null so we know this situation happened.

And when you set the pKey.setSize(0), does that cause any problems in the remaining code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added logging!

Reading the BytesWritable source, it appears that setting size to 0 is the canonical way to reset the data. The length of the actual byte array backing the BytesWritable is its capacity, which does not change.

} else {
log.warn("Received message with null message.key(): " + msgAndOffset);
pKey.setSize(0);
}

key.clear();
Expand Down