text_index

Blazing fast csv file indexing, persisting and querying

What?

This utility starts making sense when you are dealing with 20GB+ csv files, that need some manual pre-processing before moving into your database, or that you just want to tinker with.

You can build an index for any of your csv columns, as text, integer, or float type. The index will be stored on disk, typically about 3% of the original size for a text column. Parsing is performed with the excellent csv crate, processing about 800K records/sec (all threads combined) on my 2014 macbook when indexing a 64 column file.

Querying should typically be in the order of ~100 ms for equality lookups (ranges are slower). The index is stored sharded, so lookup times should not increase dramatically with input size.

Usage

Build the index

You can choose to index a column as text (str), integer (int) or floating point (float).

USAGE:
    text_index <INPUT> index <COLUMN> [TYPE]

OPTIONS:
    -t <THREADS>     Max number of THREADS
    -v               Verbose output (-v, -vv supported)

ARGS:
    <COLUMN>    Column number (starts at 1)
    <TYPE>      Type (str(default), int, float)

e.g. text_index input.csv -t 4 index 1 str

Query the index

USAGE:
    text_index <INPUT> filter <COLUMN> <OP> <VALUE> [VALUE2]

ARGS:
    <COLUMN>    Column number (starts at 1)
    <OP>        Operator (eq, lt, le, gt, ge, in, pre (starts with))
    <VALUE>     Value
    <VALUE2>    Value2 (when operator is `in`)

e.g. text_index input.csv filter 1 eq "search_string"

The future

• Support more text file formats, such as newline delimited json, or log files
• Multithreaded querying
• Support gzipped input files
• Swap friendly indexing (limit memory usage)