Skip to content
This repository has been archived by the owner on Oct 30, 2020. It is now read-only.

reset pKey when message#key is null #167

Merged
merged 2 commits into from Feb 21, 2015
Merged

reset pKey when message#key is null #167

merged 2 commits into from Feb 21, 2015

Conversation

jimjh
Copy link
Contributor

@jimjh jimjh commented Jan 27, 2015

The pKey object is reused across messages, but is not being reset between each read. #getNext should empty pKey when it receives a message without a partition key.

This is kinda tricky to reproduce in a test :( Any ideas?

The pKey object is reused across messages, but is not being reset between each read. #getNext should empty pKey when it receives a message without a partition key.
@@ -116,6 +116,8 @@ public boolean getNext(EtlKey key, BytesWritable payload, BytesWritable pKey) th
bytes = new byte[origSize];
buf.get(bytes, buf.position(), origSize);
pKey.set(bytes, 0, origSize);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually when do you see message.key() is null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It happened to us in production.

In Message.scala, one of the constructors sets key to null.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a logging when the key is null so we know this situation happened.

And when you set the pKey.setSize(0), does that cause any problems in the remaining code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added logging!

Reading the BytesWritable source, it appears that setting size to 0 is the canonical way to reset the data. The length of the actual byte array backing the BytesWritable is its capacity, which does not change.

@jimjh
Copy link
Contributor Author

jimjh commented Feb 21, 2015

Done!

HenryCaiHaiying added a commit that referenced this pull request Feb 21, 2015
reset pKey when message#key is null
@HenryCaiHaiying HenryCaiHaiying merged commit daf9686 into LinkedInAttic:master Feb 21, 2015
@HenryCaiHaiying
Copy link
Contributor

Thanks.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
2 participants