Skip to content
Browse files

Issue 20: write uncompressed bytes if uncompressed size==compressed size

The LZO specification says that we should write the uncompressed bytes
rather than the compressed bytes if the compressed buffer is actually
larger than the uncompresesd buffer.

To conform to the standard, this means we have to write the uncompressed
bytes also when they have exactly the same size as the compressed bytes.
  • Loading branch information...
1 parent 2bd0d5b commit 72b439b05407e05f9ae53aad4273244cb8eaaf7b @miguno miguno committed Apr 13, 2011
View
2 src/java/com/hadoop/compression/lzo/LzopOutputStream.java
@@ -177,7 +177,7 @@ protected void compress() throws IOException {
// the LZO specification says that we should write the uncompressed bytes rather
// than the compressed bytes. The decompressor understands this because both sizes
// get written to the stream.
- if (compressor.getBytesRead() < compressor.getBytesWritten()) {
+ if (compressor.getBytesRead() <= compressor.getBytesWritten()) {
// Compression actually increased the size of the buffer, so write the uncompressed bytes.
byte[] uncompressed = ((LzoCompressor)compressor).uncompressedBytes();
rawWriteInt(uncompressed.length);
View
10 src/test/com/hadoop/compression/lzo/TestLzopOutputStream.java
@@ -46,6 +46,7 @@
private final String bigFile = "100000.txt";
private final String mediumFile = "1000.txt";
private final String smallFile = "100.txt";
+ private final String issue20File = "issue20-lzop.txt";
@Override
protected void setUp() throws Exception {
@@ -85,6 +86,15 @@ public void testSmallFile() throws NoSuchAlgorithmException, IOException,
}
/**
+ * TODO: Write documentation.
+ *
+ */
+ public void testIssue20File() throws NoSuchAlgorithmException, IOException,
+ InterruptedException {
+ runTest(issue20File);
+ }
+
+ /**
* Test that reading an lzo-compressed file produces the same lines as reading the equivalent
* flat file. The test opens both the compressed and flat file, successively reading each
* line by line and comparing.
View
6 src/test/data/issue20-lzop.txt
@@ -0,0 +1,6 @@
+0.5 74 25425
+0.9 200 25384
+0.95 203 4
+0.98 211 2
+0.99 219 3
+0.995 240 5

0 comments on commit 72b439b

Please sign in to comment.
Something went wrong with that request. Please try again.