Skip to content
terashuf implements a quasi-shuffle algorithm for shuffling multi-terabyte text files using limited memory
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE
Makefile
README.md
terashuf.cc

README.md

terashuf

terashuf implements a quasi-shuffle algorithm for shuffling multi-terabyte text files using limited memory. It is a C++ implementation of this Python script.

Why not GNU sort -R instead?

terashuf has 2 advantages over sort -R:

  1. terashuf is much, much faster. See benchmark below.
  2. It can shuffle duplicate lines. To deal with duplicate lines in sort, the input has to be modified (append an incremental token) so that duplicate lines are different otherwise sort will hash them to the same value and place them in adjacent lines, which is not desirable in a shuffle! Then the tokens have to be removed. It's simpler to use terashuf where none of this is required.

Why not GNU shuf?

shuf does all the shuffling in-memory, which is a no-go for files larger than memory.

For small files, terashuf doesn't write any temporary files and so functions exactly like shuf. In the benchmark below, terashuf marginally outperforms shuf.

Benchmark

The following compares shuffle times for a 20GB Wikipedia dump. terashuf is tested with limited memory and with memory large enough to fit the entire file (in-memory like shuf):

Command Memory (GB) Real Time User Time Sys Time
terashuf 4 6m16s 4m53s 44s
terashuf 20 2m32s 1m17s 21s
shuf 20 2m46s 50s 27s
sort -R 4 170m51s 649m7s 52s

Benchmark run on Xeon E5-2630 @ 2.60GHz with 128GB of RAM.

Note: I'm looking for pull requests implementing the shuf interface so that terashuf can become a drop-in replacement for shuf.

Build

terashuf can be built by calling $ make. It has no dependencies other than the stdlib.

Usage

$ ./terashuf < filetoshuffle.txt > shuffled.txt

It reads 2 ENV variables:

  • TMPDIR: defaults to /tmp if not set.
  • MEMORY: defaults to 4.0, meaning use a shuffle buffer of 4 GB.

Note: the last line in the file to be shuffled will be ignored if it does not end with a newline marker (\n).

When shuffling very large files, terashuf needs to keep open SIZE_OF_FILE_TO_SHUFFLE / MEMORY temporary files. Make sure to set the maximum number of file descriptors to at least this number. By setting a large file descriptor limit, you ensure that terashuf won't abort a shuffle midway, saving precious researcher time.

Quasi-shuffle

terashuf implements a quasi-shuffle as follows:

  1. Divide N input lines into K files containing L lines.
  2. Shuffle each of the K files (this is done in memory before writing the file).
  3. Read one line from each of the K files into a buffer until the buffer has L lines.
  4. Shuffle the buffer and write to output.
  5. Repeat 3. and 4. until all lines have been written to output.

TODO

Pull requests are very welcome!

  • Rather than use fixed-length lines in the buffer which wastes memory, use variable-length lines such that all buffer memory is used.
  • Implement --help
  • Implement shuf interface so that terashuf becomes a drop-in replacement
  • Add benchmarks

License

Copyright (c) 2017 Salle, Alexandre atsalle@inf.ufrgs.br. All work in this package is distributed under the MIT License.

You can’t perform that action at this time.