add LOG.warn() about insufficiently large io.compression.codec.lzo.buffersize #61

Merged
merged 3 commits into from Feb 25, 2013

2 participants

@ekoontz

Addresses #60.

@rangadi
Twitter, Inc. member

This probably explains the failures we saw when we enabled Pig's temp file compression. Thanks.

@rangadi rangadi commented on an outdated diff Feb 23, 2013
src/java/com/hadoop/compression/lzo/LzoDecompressor.java
@@ -201,6 +201,16 @@ public LzoDecompressor() {
}
public synchronized void setInput(byte[] b, int off, int len) {
+ if (!isCurrentBlockUncompressed()) {
+ if (len > directBufferSize) {
+ LOG.warn("Decompression will fail because compressed buffer size :" +
+ len + " is greater than this decompressor's directBufferSize: " +
+ directBufferSize + ". To fix this, You can increase the value of your " +
+ "configuration's io.compression.codec.lzo.buffersize to be larger " +
+ "than: " + len + " and restart your job.");
@rangadi
Twitter, Inc. member
rangadi added a note Feb 23, 2013

can you remove " and restart your job"? The codec is outside of MR jobs as well. May be better to rephrase it to remove the first "You " as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ekoontz

Thanks for the comments rangadi.

@rangadi rangadi merged commit 0c07467 into twitter:master Feb 25, 2013
@ekoontz ekoontz referenced this pull request Feb 25, 2013
Closed

fix typo in last commit #62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment