Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: parallel io.Copy #664

Closed
mmatczuk opened this issue Jan 29, 2024 · 1 comment
Closed

perf: parallel io.Copy #664

mmatczuk opened this issue Jan 29, 2024 · 1 comment
Assignees
Labels
Milestone

Comments

@mmatczuk
Copy link
Contributor

The io.Copy implementation is synchronous. It reads and writes in a for loop on a single thread. When read and write costs are high (i.e. using net.Conn) it cal halve the performance.

We could easily optimize that by prefetching data while writing.

Now

R R R
 W W W

Desired

RRR
 WWW

Implementation idea:

  • Reader uses two buffers 128K each
  • If buffer capacity goes below 32kiB the buffers are switched
  • Reader sends buffer subslices to writer over a channel, channel capacity is 1
  • Writer writes what it gets over the channel as in io.Copy

Note use subslice with capacity [start:len:cap] syntax.

@mmatczuk mmatczuk modified the milestones: v1.2, v1.3 Jan 29, 2024
@mmatczuk mmatczuk added the perf label Feb 2, 2024
@mmatczuk mmatczuk self-assigned this Jun 5, 2024
@mmatczuk
Copy link
Contributor Author

The idea was implemented in #834 but did not yield the expected results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant