The FileBasedCollection is a Java Colelction backed by a file. The file is a binary, compressed file, generated with the Snappy library. The version used here is snappy-java-1.1.2.1.jar.
The FileBasedCollection is a modifiable collection that extends the AbstractCollection and implements Closeable. The FileBasedCollection has an internal store and a chunk size. If the internal store contains more elements then the chunk size, the internal store is swapped to disk. The iterator implements Closeable and runs through the file backing the collection, loading the next n elements. The iterator is auto-closeable. Great care has been taken to clean up data after the iterator, including shut-down hooks, but if the process crashes, the backing file will be left over. This collection is meant for very large collections that can't possibly fit in memory. It has been tested with billions of items and several gigabytes of data. While using the FileBasedCollection during testing, on several occasions the disk was full and needed cleaning. During production the FileBasedCollection proved stable, even under stressful conditions. The FileBasedColelction has been used with external sorting where many instances where created and merged. The performance and memory consumption was very acceptable. The FileBasedCollection can only grow by adding elements to the head. This limits its use to applications where an iterator is sufficient.