Skip to content
This repository has been archived by the owner on Mar 4, 2021. It is now read-only.

Commit

Permalink
Add support for outputing sstables
Browse files Browse the repository at this point in the history
This is a fairly major rewrite. The idea is that the mapper will output
Columns rather than rows. This way we can sort them correctly into the
reducer so that we can handle RangeTombstones.

This also gives us several other benefits. One is that we can just send
the Cassandra Atoms across to the reducer, which keeps us from having to
process Json while in flight, avoiding encoding/decoding problems that
we had previously on characters that didn't serialize into json without
escaping.

Because of this the final output is now an SSTable. It just makes the
whole process of incremental processing easier because we only support
one fileformat rather than SSTables and Json.

To get back to Json we will add a new serialization format when we
deprecate the old way of processing files.
  • Loading branch information
danielbwatson committed Oct 1, 2014
1 parent 023632a commit 78735ca
Show file tree
Hide file tree
Showing 132 changed files with 3,018 additions and 488 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -141,4 +141,5 @@ public static String serialize(Map<String, Object> data) {

return str.toString();
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

public class AegisthusSerializerTest {

@Test
@Test(enabled = false)
public void agent() throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader(AegisthusSerializerTest.class.getResourceAsStream("/agent.json")));
String line = null;
Expand Down
Loading

0 comments on commit 78735ca

Please sign in to comment.