S3 files corruption

[estezz]

Secor is leaving the end of messages in the output of smaller messages.

If you load these 2 messages into kafka.
{"array":[1,2,3],"boolean":true,"null":null,"dog":"brown","dog2":"brown","dog3":"brown","dog4":"brown","dog5":"brown","dog6":"brown","dog7":"brown","dog8":"brown","dog9":"brown","dog10":"brown","dog11":"brown","number":123,"object":{"a":"b","c":"d","e":"f"},"string":"Hello World"}

{"array":[1,2,3],"boolean":true,"null":null,"number":123,"object":{"a":"b","c":"d","e":"f"},"string":"Hello World"}

The secor file in S3 will look like this. I put the extra data in bold on line 14
13: {"array":[1,2,3],"boolean":true,"null":null,"dog":"brown","dog2":"brown","dog3":"brown","dog4":"brown","dog5":"brown","dog6":"brown","dog7":"brown","dog8":"brown","dog9":"brown","dog10":"brown","dog11":"brown","number":123,"object":{"a":"b","c":"d","e":"f"},"string":"Hello World"}

14: {"array":[1,2,3],"boolean":true,"null":null,"number":123,"object":{"a":"b","c":"d","e":"f"},"string":"Hello World"}n","dog6":"brown","dog7":"brown","dog8":"brown","dog9":"brown","dog10":"brown","dog11":"brown","number":123,"object":{"a":"b","c":"d","e":"f"},"string":"Hello World"}

Looks like this is an issue with BytesWritable. Here is some code testing that.
package secor;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.SequenceFile;

public class TestWriter {

private  SequenceFile.Writer mWriter;


public static void main(String[] args) {

    TestWriter testWriter = new TestWriter();

    try {
        testWriter.init();

        testWriter.write(10, "This is the shit");
        testWriter.write(11, "Leo");
        testWriter.mWriter.close();

        testWriter.read();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (InstantiationException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IllegalAccessException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


}

public void init() throws IOException{
    Configuration config = new Configuration();
    FileSystem fs = FileSystem.get(config);
    Path fsPath = new Path("/tmp/testfile");

    this.mWriter = SequenceFile.createWriter(fs, config, fsPath,
        LongWritable.class, BytesWritable.class);
}


public void write(long key, String value) throws IOException {

    LongWritable writeableKey = new LongWritable(key);
    BytesWritable writeableValue = new BytesWritable(value.getBytes());
    System.out.println(new String(writeableValue.getBytes()));
    this.mWriter.append(writeableKey, writeableValue);

}

public void read() throws IOException, InstantiationException, IllegalAccessException{
    Configuration config = new Configuration();
    FileSystem fs = FileSystem.get(config);
    Path fsPath = new Path("/tmp/testfile");

    SequenceFile.Reader reader = new SequenceFile.Reader(fs, fsPath, new Configuration());
    LongWritable key = new LongWritable();
    BytesWritable value = new BytesWritable();
    System.out.println("reading file " + fsPath);
    while (reader.next(key, value)) {
        System.out.println(new String(value.getBytes()));
    }
}

}

[estezz]

This apears to be a ptoblem with the com.pinterest.secor.main.LogFilePrinterMain instead of the writing of the file. I will close this and open a new one for the com.pinterest.secor.main.LogFilePrinterMain

[foovungle]

Has this been fixed? Thx

[estezz]

Yes this is fixed. It was a tool bug not a data problem. Look at my comment on Jan 14

[foovungle]

Got it. Thx