Fastest way to read a csv file?

[patricebellan]

Hi,

I've been "playing" with xsv for a couple days and was please to see it's the fasted csv "kit" I've tried.
So I decided to give Rust and the rust-csv crate a try.

Picking code from the examples, I came up with a simple function that iterates over a file and counts the rows.

It takes about 25 seconds to run on a 7.2M rows file, which is fine.
But I was surprised it's exactly the time it takes xsv to run the following "select" on the same file (no prior "index" was done).

time xsv select -d';' field myfile.csv > field.txt
real    0m24.803s
user    0m9.176s
sys 0m9.336s

I was expecting my loop to run faster for it does pretty much nothing, so I'm thinking I probably missed something (once again, I'm a total Rust beginner).

Here's the code:

extern crate csv;

fn main() {
    let mut cnt = 0;
    let mut rdr = csv::Reader::from_file("myfile.csv").unwrap().delimiter(b';');

    for row in rdr.records() {
        cnt+=1;
    }

    println!("{}",cnt);
}

Is there any faster way to read the file?

Note: I'm not complaining about speed, I'm just surprised xsv can do much more than my simple piece of code in the same time ;)

[Eh2406]

Did you compile in release? Don't know the real answer, just a knee jerk response to "Rust beginner" and "expecting to run faster" :-)

[BurntSushi]

@patricebellan Have you read the section in the docs about iterating over records?

I would somewhat expect xsv to run quite a bit faster than 25 seconds on a mere 7 million rows. I share @Eh2406's concerns. Instead of cargo build you might try cargo build --release.

xsv select barely does anything either. It should run within spitting distance of a simple count loop.

[patricebellan]

Nice try @Eh2406, I did compile in release ;)

I'm running it on a VM, so overall performance may not be the best.
But I was mostly concerned about comparing both, not pure performance per se.

[hakunin]

Just ran into this myself - debug mode reads mere 2M rows in 18seconds, release build does it in 630ms.

[phiresky]

Just in case someone else finds this, here's an overview of things to make it faster (tested on a file with four columns and one billion lines):

1. compile with --release (huge difference, >10x perf)

2. wrap your input in a BufReader::with_capacity(1_000_000). No difference for me, probably depends on where the data comes from

3. use .byte_records instead of .records if you don't need string parsing (only minor difference for me)

4. enable this: (opt-level 3 is not much faster than level 2, lto=fat improves perf by 15%!)

    [profile.release]
    opt-level = 3
    debug = true
    lto = "fat"
   

5. compile with RUSTFLAGS="-C target-cpu=native" (only minor difference)

6. Instead of for result in reader.into_byte_records(), use:

   let mut record = csv::ByteRecord::new();
   while reader.read_byte_record(&mut record)? {

   this doubles the performance!! This is also what xsv does: https://github.com/BurntSushi/xsv/blob/3de6c04269a7d315f7e9864b9013451cd9580a08/src/cmd/select.rs#L77

With the release mode and read_byte_record, perf of using the library is the same as xsv select for me.

[BurntSushi]

Thanks for adding your tip here! There is more explanation here on the technique: https://docs.rs/csv/1.1.3/csv/tutorial/index.html#amortizing-allocations

[phiresky]

Thanks for the link. Funny, I didn't actually see that, I just searched for "performance" within the docs index page but it didn't have results, and I assumed "tutorial" was more about handling different types of files etc, not about improving performance. And the docs search (obviously i guess) didn't yield it either :)