Q: Does num_workers process the file in parallel?

[cdesch]

Just as the title says, if I want to process a file with the rows processing independently of each other (non-sequential), does setting the num_workers to, let's say 6, process the file in 6 independent processes?

iex> "../test/fixtures/docs/valid.csv"
iex> |> Path.expand(__DIR__)
iex> |> File.stream!
iex> |> CSV.decode(num_workers: 6)

[beatrichartz]

It splits the stream into 6 parallel processes. The File.stream! itself is not parallelised, but is rarely the bottleneck.

[cdesch]

So that is only allocating 6 processes to handle the work of that one File.stream!?

Would I need to run 6 separate tasks of File.stream! on 6 different files with CSV.decode(num_workers: 6) to get 36 processes?

[beatrichartz]

I will close this for now as the File Stream should not be the bottleneck in this. If there is a performance or backpressure issue that you're seeing in your setup please feel free to reopen.