csvsample
extracts some rows from CSV file to create randomly sampled CSV.
- The size of original file does not need to be specified beforehand. It means that the sampling process can be applied to data stream with unknown size such as system logs, no matter how large the amount of data is.
- All methods accepts optional
seed
value. The same data set with the same sampling rate always yields exactly the same result, which is good for reproducibility.
You can install csvsample
via pip
:
pip install csvsample
csvsample.sample()
is the main API:
csvsample.sample(lines, sampling_method, **kwargs)
lines
can be any iterable
containing valid CSV rows including header
row.
sampling_method
should be one of followings:
random
hash
reservoir
random
sampling method performs random sampling using pseudo random number
generator:
import csvsample
with open('input.csv', 'r') as i:
with open('output.csv', 'w') as o:
o.writelines(csvsample.sample(i, 'random', sample_rate=0.1))
hash
sampling method performs hash-based sampling using extremely-fast hash
function.
Let's say that instead of saving all users' log, you want to randomly select 10% of users and only save logs of those selected users. Simple random sampling won't work. You can use hash-based sampling. "Consistent" nature of the algorithm guarantees that any user ID selected once will always be selected again:
sampled = csvsample.sample(lines, 'hash', sample_size=0.1, col='user_id')
reservoir
sampling method performs reservoir sampling. Let's say that you
have an URL of 100GB csv file. Since you don't have enough disk space, you just
want to save small portion of sample which is representative and unbiased.
reservoir sampling method allows you to acquire random sample without saving entire data first:
sampled = csvsample.sample(lines, 'reservoir', sample_size=1000)
Now sampled
variable contains exactly 1,000 randomly selected lines.
There are some convenience helpers:
csvsample.sample_url(url, sampling_method, **kwargs)
read CSV from givenurl
. You can specify character set encoding viaencoding
keyword argument. (default:utf-8
)
csvsample.sample()
and other helpers return a generator containing sampled
CSV rows including header row. The generator contains special function
to_buf()
which converts itself into io.StringIO
instance so that you can
pass the sampled CSV to other libraries such as Pandas:
import csvsample
import pandas as pd
sampled = csvsample.sample_url(url, 'random', sample_rate=0.1)
df = pd.read_csv(sampled.to_buf())
csvsample
also provides command-line interface.
Following URL contains a CSV file from DataHub:
> curl -sL https://bit.ly/2ItnHvK | head
region,year,population
WORLD,1950,2536274.721
WORLD,1951,2583816.786
WORLD,1952,2630584.384
WORLD,1953,2677230.358
A number of rows including header is 18019:
> curl -sL https://bit.ly/2ItnHvK | wc -l
18019
Let's make 10% of random sample:
> curl -sL https://bit.ly/2ItnHvK | csvsample random 0.1 > sample.csv
> wc -l sample.csv
1777 sample.csv
> head -n 5 sample.csv
region,year,population
WORLD,1952,2630584.384
WORLD,1972,3851545.181
WORLD,1977,4229201.257
WORLD,1988,5148556.956
You may use reservoir sampling method to obtain exact number of rows:
> curl -sL https://bit.ly/2ItnHvK | csvsample reservoir 100 > sample.csv
> wc -l sample.csv
100 sample.csv