Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split the data processor into a number of parallel workers #58

Open
anatoly-scherbakov opened this issue Oct 15, 2020 · 0 comments
Open

Comments

@anatoly-scherbakov
Copy link
Collaborator

The CSV data processor is right now single-threaded; it processes the lines of incoming data stream one by one, sequentially.

This was acceptable for a proof of concept version, but now we should spawn a number of parallel threads instead. Every thread will work independently of the others.

Every row of the input will be dispatched to those worker threads via channels. When processing is complete, via another channel the output row will be sent to the writer thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant