Unzip the file after downloading

[honzakral]

the bz2 uncompress is eating some CPU so it's not stressing elasticsearch as much as possible, the observed difference is small, but it is there...

[mikemccand]

++

I also say Python chewing ~100% CPU, which is no good :) When I run on an already uncompressed docs source it's more like 15% CPU.

[honzakral]

btw if we don't want to spend the disk space maybe it would be an option to spawn an external process for the unzipping (bunzip2 --stdout) and read from the pipe.

this can be done either via multiprocessing package or subprocess.

[danielmitterdorfer]

I am really wary of generating any additional load while running the benchmark to avoid skewing results so I'd rather spend the additional disk space. tbh I'd rather physically separate also the client from the cluster under test but it takes some time until we get there (#25). So I'm +1 on unzipping before-hand.

[mikemccand]

Maybe a good "goldilocks" solution would be to use .gz instead of .bz2: it will be a bit larger, but CPU cost for decompression is much lower.