New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cookbook ideas for threadpool crate #225

Open
ericho opened this Issue Jul 7, 2017 · 10 comments

Comments

Projects
None yet
6 participants
@ericho
Contributor

ericho commented Jul 7, 2017

Issue to gather ideas for the threadpool crate

edit:

@KodrAus

This comment has been minimized.

Show comment
Hide comment
@KodrAus

KodrAus Jul 12, 2017

Dispatching some blocking io to a threadpool and returning a Result on a channel?

KodrAus commented Jul 12, 2017

Dispatching some blocking io to a threadpool and returning a Result on a channel?

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Jul 13, 2017

Collaborator
  • Generate thumbnails in parallel - use walkdir to find all image files in subdirs and create scaled down thumbnails next to originals with image crate. Any other interesting file processing might do here in fact.
  • Gather output from multiple child processes - Manage a pool of child processes doing some IO bound work (like a list of git clone addresses)
  • Process memory mapped file in parallel - memmap a csv (or any easily divisible) file and process the chunks against a transform (csv reader/regex filter/regex replace) in a threadpool. Chunking here might be tricky. Depending on the example the results might need to be sorted afterwards.
Collaborator

budziq commented Jul 13, 2017

  • Generate thumbnails in parallel - use walkdir to find all image files in subdirs and create scaled down thumbnails next to originals with image crate. Any other interesting file processing might do here in fact.
  • Gather output from multiple child processes - Manage a pool of child processes doing some IO bound work (like a list of git clone addresses)
  • Process memory mapped file in parallel - memmap a csv (or any easily divisible) file and process the chunks against a transform (csv reader/regex filter/regex replace) in a threadpool. Chunking here might be tricky. Depending on the example the results might need to be sorted afterwards.
@yandexx

This comment has been minimized.

Show comment
Hide comment
@yandexx

yandexx Jul 24, 2017

Contributor
  • Calculating checksum for all files (or by a glob) in a folder; this task is commonly CPU-bound.
Contributor

yandexx commented Jul 24, 2017

  • Calculating checksum for all files (or by a glob) in a folder; this task is commonly CPU-bound.
@ericho

This comment has been minimized.

Show comment
Hide comment
@ericho

ericho Aug 22, 2017

Contributor

Just extending @yandexx suggestion, an example could be calculate the checksum of all files in a folder tree using walkdir.

Contributor

ericho commented Aug 22, 2017

Just extending @yandexx suggestion, an example could be calculate the checksum of all files in a folder tree using walkdir.

@budziq budziq referenced this issue Aug 31, 2017

Closed

Cookbook ideas for rayon #272

6 of 6 tasks complete

@budziq budziq changed the title from Ideas for threadpool crate to Cookbook ideas for threadpool crate Sep 28, 2017

@budziq budziq added this to the impl period milestone Oct 4, 2017

@iamdanfox

This comment has been minimized.

Show comment
Hide comment
@iamdanfox

iamdanfox Oct 13, 2017

Trying to think of non-file system traversing examples here... what about overlaying some text onto an animated gif? Each frame in the gif format could be processed completely independent and the piston.rs/image/gif crate can already encode/decode nicely.

iamdanfox commented Oct 13, 2017

Trying to think of non-file system traversing examples here... what about overlaying some text onto an animated gif? Each frame in the gif format could be processed completely independent and the piston.rs/image/gif crate can already encode/decode nicely.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Oct 27, 2017

Collaborator

@iamdanfox I'm not sure if this idea is significantly different form the fractal drawing example (which is a little artificial in itself). Could you point how this example would be different (except for per frame vs per pixel parallelism) or maybe you have some other cool ideas?

Collaborator

budziq commented Oct 27, 2017

@iamdanfox I'm not sure if this idea is significantly different form the fractal drawing example (which is a little artificial in itself). Could you point how this example would be different (except for per frame vs per pixel parallelism) or maybe you have some other cool ideas?

@AndyGauge AndyGauge referenced this issue Apr 20, 2018

Closed

Tracking Issues #396

13 of 13 tasks complete
@AndyGauge

This comment has been minimized.

Show comment
Hide comment
@AndyGauge

AndyGauge Apr 23, 2018

Collaborator

I'd like to expand the examples in threadpool to do something Io bound that would be beneficial to a wider audience.

Collaborator

AndyGauge commented Apr 23, 2018

I'd like to expand the examples in threadpool to do something Io bound that would be beneficial to a wider audience.

@budziq budziq removed this from the impl period milestone May 16, 2018

@AndyGauge

This comment has been minimized.

Show comment
Hide comment
@AndyGauge

AndyGauge Jul 18, 2018

Collaborator

Maybe we could check for clients listening on port 80. Say a request to github.io in each thread to verify the port is responding. I'm thinking using essentially this: https://doc.rust-lang.org/std/net/struct.TcpStream.html#examples to check each of these addresses: 185.199.X.153 where X is 108,109,110,111.

Collaborator

AndyGauge commented Jul 18, 2018

Maybe we could check for clients listening on port 80. Say a request to github.io in each thread to verify the port is responding. I'm thinking using essentially this: https://doc.rust-lang.org/std/net/struct.TcpStream.html#examples to check each of these addresses: 185.199.X.153 where X is 108,109,110,111.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Jul 20, 2018

Collaborator

Maybe we could check for clients listening on port 80.

This seams a little artificial. I think that original approach of "we have a list of crates and what can we do with these" might be suboptimal in some cases (especially relatively low level crates such as this one). How about we post issues asking for contributors ideas for given domain instead of given crate?

Collaborator

budziq commented Jul 20, 2018

Maybe we could check for clients listening on port 80.

This seams a little artificial. I think that original approach of "we have a list of crates and what can we do with these" might be suboptimal in some cases (especially relatively low level crates such as this one). How about we post issues asking for contributors ideas for given domain instead of given crate?

@KodrAus

This comment has been minimized.

Show comment
Hide comment
@KodrAus

KodrAus Jul 23, 2018

@budziq Focusing on domains rather than specific libraries sounds like a good approach to me and probably more in line with the new organization too.

KodrAus commented Jul 23, 2018

@budziq Focusing on domains rather than specific libraries sounds like a good approach to me and probably more in line with the new organization too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment