Concurrent IO reader package

A small package to split a binary stream (well, a file) into multiple chunks for concurrent processing. Depending on the stream it may even be parallel I/O, but it's certainly concurrent. I use this for a few things internally where I need/want parallel I/O reading.

How it works

Since an io.ReaderAt can be read concurrently this lib splits the underlying stream into chunks with a io.SectionReader. Those can then be sent off to other goroutines and read concurrently. See reader_test.go as an example on how to use it.

There's also a Hash() method to hash the chunks independently (and concurrently).

Tests

To run the tests first generate a test file with

# First generate file
$ go generate
1025+0 records in
1025+0 records out
524800 bytes (525 kB, 512 KiB) copied, 0.0191943 s, 27.3 MB/s
# Then test
$ go test -race -v

Author

Mathias Seiler, mathias.seiler@mironet.ch