New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cookbook ideas for rayon #272

Closed
KodrAus opened this Issue Aug 24, 2017 · 15 comments

Comments

Projects
None yet
8 participants
@KodrAus

KodrAus commented Aug 24, 2017

Come up with ideas for nice introductory examples of using the rayon crate, possibly in combination with other crates, that would be good to show in the Rust Cookbook. Please leave a comment here with your ideas! You don't necessarily have to write the example code yourself but PRs are always welcome!

rayon already has a suite of examples but they seem mostly focused on benchmarking.

Edit:


@LeoTindall

This comment has been minimized.

Show comment
Hide comment
@LeoTindall

LeoTindall Aug 29, 2017

I wrote a Fizz Buzz example which might be appropriate: https://gist.github.com/LeoTindall/2bed40527d3920a73e9054fc80876b9c

It's (conveniently) fairly congruent with my Go example, which could be useful for those moving along the Go-to-Rust sidegrade path (https://gist.github.com/LeoTindall/433654da49c93c2da347bb7b5f7adc79).

LeoTindall commented Aug 29, 2017

I wrote a Fizz Buzz example which might be appropriate: https://gist.github.com/LeoTindall/2bed40527d3920a73e9054fc80876b9c

It's (conveniently) fairly congruent with my Go example, which could be useful for those moving along the Go-to-Rust sidegrade path (https://gist.github.com/LeoTindall/433654da49c93c2da347bb7b5f7adc79).

@lilianmoraru

This comment has been minimized.

Show comment
Hide comment
@lilianmoraru

lilianmoraru Aug 31, 2017

Contributor

I would like to see the rayon + features integration, but I guess this will turn into bigger, multi-line examples which might not be the most appropriate for the cookbook(don't know).

Contributor

lilianmoraru commented Aug 31, 2017

I would like to see the rayon + features integration, but I guess this will turn into bigger, multi-line examples which might not be the most appropriate for the cookbook(don't know).

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Aug 31, 2017

Collaborator

Hi @LeoTindall

Thanks for you interest! We would like to keep the cookbook focused on examples that can be considered:

  1. working code snippet realizing some real life use-case
  2. commonly needed building block that programmers might need

So FizzBuzz (parallel or not) might not be the best fit. Would you be able to think of an example that would fit into one of these categories?
You might consider using some of the ideas posted for threadpool

might not be the most appropriate for the cookbook(don't know).

@lilianmoraru - not necessarily. Do you have anything concrete in mind?

We might consider rewriting @ericho 's PR to use rayon (unfortunately dropping walkdir). Currently we have 2 super simple rayon examples and single relatively advanced threadpool one. I guess we should go with the one most ergonomic for the job .

Collaborator

budziq commented Aug 31, 2017

Hi @LeoTindall

Thanks for you interest! We would like to keep the cookbook focused on examples that can be considered:

  1. working code snippet realizing some real life use-case
  2. commonly needed building block that programmers might need

So FizzBuzz (parallel or not) might not be the best fit. Would you be able to think of an example that would fit into one of these categories?
You might consider using some of the ideas posted for threadpool

might not be the most appropriate for the cookbook(don't know).

@lilianmoraru - not necessarily. Do you have anything concrete in mind?

We might consider rewriting @ericho 's PR to use rayon (unfortunately dropping walkdir). Currently we have 2 super simple rayon examples and single relatively advanced threadpool one. I guess we should go with the one most ergonomic for the job .

@nocduro

This comment has been minimized.

Show comment
Hide comment
@nocduro

nocduro Sep 1, 2017

Contributor

Rust/git noob here. I recently made a basic parallel image resizer with rayon here. I mostly used it to learn some more about rayon, so it is quite basic and only resizes images at the same directory level (not going into sub directories).

The hardest part for me was figuring out how to handle the errors, since it's not possible to use try! or ? on parallel code and as far as I saw none of the previous examples call a function that returns a Result. I'm still not sure if I'm doing the error handling correctly, so if anyone with more experience knows something better, let me know!

Contributor

nocduro commented Sep 1, 2017

Rust/git noob here. I recently made a basic parallel image resizer with rayon here. I mostly used it to learn some more about rayon, so it is quite basic and only resizes images at the same directory level (not going into sub directories).

The hardest part for me was figuring out how to handle the errors, since it's not possible to use try! or ? on parallel code and as far as I saw none of the previous examples call a function that returns a Result. I'm still not sure if I'm doing the error handling correctly, so if anyone with more experience knows something better, let me know!

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Sep 1, 2017

Collaborator

That is a very nice example @nocduro!
Are you willing to submit it as a PR? There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way! 👍

Collaborator

budziq commented Sep 1, 2017

That is a very nice example @nocduro!
Are you willing to submit it as a PR? There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way! 👍

@nocduro

This comment has been minimized.

Show comment
Hide comment
@nocduro

nocduro Sep 2, 2017

Contributor

Are you willing to submit it as a PR?

Sure! I've never made a pull request before, but I should be able to figure it out. I just read the CONTRIBUTING.md file as well.

There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way!

Awesome, thank you!

Contributor

nocduro commented Sep 2, 2017

Are you willing to submit it as a PR?

Sure! I've never made a pull request before, but I should be able to figure it out. I just read the CONTRIBUTING.md file as well.

There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way!

Awesome, thank you!

@budziq budziq added this to the impl period milestone Oct 4, 2017

@danielpclark

This comment has been minimized.

Show comment
Hide comment
@danielpclark

danielpclark Oct 8, 2017

I think I have a good example idea for the Rayon Book example. I thought how I might be able to use Rayon for a regular iterator and it comes down to this.

First create a chunk builder method which will fill a vector of a certain quantity of work to work on. Then use Rayon's par_iter over each chunk. So any iterator can act as a Vec collection with Rayon if you chunk out the work.

For chunk size I went with 128 items or (in my new version) 32 times however many cores there are. I loop the chunks of work until I get my desired result. To see the code I have this implemented in here's an earlier version of my code that's simpler to look at: chunk_sequence and its use with par_iter. My project is a brute force password attempter for encrypted files with AES encryption. These code snippets shouldn't be hard to understand to base an example off of.

danielpclark commented Oct 8, 2017

I think I have a good example idea for the Rayon Book example. I thought how I might be able to use Rayon for a regular iterator and it comes down to this.

First create a chunk builder method which will fill a vector of a certain quantity of work to work on. Then use Rayon's par_iter over each chunk. So any iterator can act as a Vec collection with Rayon if you chunk out the work.

For chunk size I went with 128 items or (in my new version) 32 times however many cores there are. I loop the chunks of work until I get my desired result. To see the code I have this implemented in here's an earlier version of my code that's simpler to look at: chunk_sequence and its use with par_iter. My project is a brute force password attempter for encrypted files with AES encryption. These code snippets shouldn't be hard to understand to base an example off of.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Oct 8, 2017

Collaborator

Thanks @danielpclark the technique is very interesting! Although we would like to focus on solving common real-life programming problems (brute decryption might not be all that common usecase). Would you be able to formulate the idea in terms of a programming task to solve?

Collaborator

budziq commented Oct 8, 2017

Thanks @danielpclark the technique is very interesting! Although we would like to focus on solving common real-life programming problems (brute decryption might not be all that common usecase). Would you be able to formulate the idea in terms of a programming task to solve?

@danielpclark

This comment has been minimized.

Show comment
Hide comment
@danielpclark

danielpclark Oct 8, 2017

@budziq How about don't call it brute forcing a password. Call it a game of guess the number. Otherwise I have no common use case.

danielpclark commented Oct 8, 2017

@budziq How about don't call it brute forcing a password. Call it a game of guess the number. Otherwise I have no common use case.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Oct 9, 2017

Collaborator

@danielpclark guessing the number might not be an ideal use case but I'm sure there is a nice one somewhere in there but I cannot point my finger on it. If you come up with something that could be useful to a programmer trying to get things done see our inclusion policy draft please post it right away!

Collaborator

budziq commented Oct 9, 2017

@danielpclark guessing the number might not be an ideal use case but I'm sure there is a nice one somewhere in there but I cannot point my finger on it. If you come up with something that could be useful to a programmer trying to get things done see our inclusion policy draft please post it right away!

@danielpclark

This comment has been minimized.

Show comment
Hide comment
@danielpclark

danielpclark Oct 9, 2017

@budziq I've been putting a lot of thought into what you've asked for “common real-life programming problems”, “an ideal use case”, and “commonly needed building block that programmers might need”.

I've come to the conclusion that in general threading isn't common to many people and has more niche applications. The code example I've provided could easily by used for mathematical computations, or processing large collections of data through filters/tasks/jobs.

The problem I'm running in to is your question is a general programming question but the audience is too broad. If this were the R programming language I would have a better focus on who the examples were to be made for since they are a niche that focus on math, statistics, and learning algorithms. I don't know what the exact makeup of the Rust community is so I don't know which particular applications to design an example for.

The particular design for my example is as follows. We have a target variable which will hold the one answer the system is looking for in a Mutex. This doesn't have to be a one answer kind of thing, it can hold a Vector and separate threads can each call for a lock on it when they have something of relevance to put in it and build the data. Secondly my example has the loop to process input data in chunks and that is great for keeping track of performance and analytics as progress is being made. The par_iter is the no brainer part of Rayon which is super cool to use and just gets the job done. Beyond that this way of using Rayon has many potential applications and can be modified for any example as you see fit.

If you want me to produce an example I need to know the target audience. In my day to day life I don't process large amounts of anything and most of my work is sequential and trivial. So I don't commonly have much use of threads and I'm not in the habit of thinking of how to work in parallel. The other examples I might think of for web application doesn't work with pre-existing data but rather waits to spin up threads and that would be an entirely different example with a different implementation.

Without knowing the target audience I'm not sure any real “common, or ideal, use case” will arise.

danielpclark commented Oct 9, 2017

@budziq I've been putting a lot of thought into what you've asked for “common real-life programming problems”, “an ideal use case”, and “commonly needed building block that programmers might need”.

I've come to the conclusion that in general threading isn't common to many people and has more niche applications. The code example I've provided could easily by used for mathematical computations, or processing large collections of data through filters/tasks/jobs.

The problem I'm running in to is your question is a general programming question but the audience is too broad. If this were the R programming language I would have a better focus on who the examples were to be made for since they are a niche that focus on math, statistics, and learning algorithms. I don't know what the exact makeup of the Rust community is so I don't know which particular applications to design an example for.

The particular design for my example is as follows. We have a target variable which will hold the one answer the system is looking for in a Mutex. This doesn't have to be a one answer kind of thing, it can hold a Vector and separate threads can each call for a lock on it when they have something of relevance to put in it and build the data. Secondly my example has the loop to process input data in chunks and that is great for keeping track of performance and analytics as progress is being made. The par_iter is the no brainer part of Rayon which is super cool to use and just gets the job done. Beyond that this way of using Rayon has many potential applications and can be modified for any example as you see fit.

If you want me to produce an example I need to know the target audience. In my day to day life I don't process large amounts of anything and most of my work is sequential and trivial. So I don't commonly have much use of threads and I'm not in the habit of thinking of how to work in parallel. The other examples I might think of for web application doesn't work with pre-existing data but rather waits to spin up threads and that would be an entirely different example with a different implementation.

Without knowing the target audience I'm not sure any real “common, or ideal, use case” will arise.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Oct 10, 2017

Collaborator

@danielpclark Thanks for an explanation!

So I don't commonly have much use of threads

No worries and no pressure, neither do I. That is why there are so little concurrency examples to date ;)

I need to know the target audience.

It will be an extremely hard question to answer. Rust being general purpose language attracts users from a very broad spectrum (webdev, embedded, server, and many more, recently even some HPC folks started to take interest).

The best we can do here is to gather questions from the community in the form
"how am I to do XYZ with Rust and available crates" and most of the time we are acting as the source for these questions.

Rayon is especially hard in this matter as it magically just works. Most likely it would be conductive to some basic map-reduce scheme like the one written with std in RBE. Another approach would be to go over rayon's docs and search for inspiration while browsing its (quite large) feature-set.

Collaborator

budziq commented Oct 10, 2017

@danielpclark Thanks for an explanation!

So I don't commonly have much use of threads

No worries and no pressure, neither do I. That is why there are so little concurrency examples to date ;)

I need to know the target audience.

It will be an extremely hard question to answer. Rust being general purpose language attracts users from a very broad spectrum (webdev, embedded, server, and many more, recently even some HPC folks started to take interest).

The best we can do here is to gather questions from the community in the form
"how am I to do XYZ with Rust and available crates" and most of the time we are acting as the source for these questions.

Rayon is especially hard in this matter as it magically just works. Most likely it would be conductive to some basic map-reduce scheme like the one written with std in RBE. Another approach would be to go over rayon's docs and search for inspiration while browsing its (quite large) feature-set.

@j-haj

This comment has been minimized.

Show comment
Hide comment
@j-haj

j-haj Oct 12, 2017

Contributor

Any interest in parallel search using findy_any or any? The examples may be trivially small but nonetheless I think useful. If there is interest, I am happy to work on this.

Contributor

j-haj commented Oct 12, 2017

Any interest in parallel search using findy_any or any? The examples may be trivially small but nonetheless I think useful. If there is interest, I am happy to work on this.

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Oct 12, 2017

Collaborator

@j-haj I'd say certainly yes on both accounts! If I were to showcase any id also show all. 👍

Just propose the working titles of both examples and we are ready to go. I'll create the issues.


Edit:
Already created the issue! Thanks for jogging me into this @j-haj and @danielpclark !

Collaborator

budziq commented Oct 12, 2017

@j-haj I'd say certainly yes on both accounts! If I were to showcase any id also show all. 👍

Just propose the working titles of both examples and we are ready to go. I'll create the issues.


Edit:
Already created the issue! Thanks for jogging me into this @j-haj and @danielpclark !

@AndyGauge

This comment has been minimized.

Show comment
Hide comment
@AndyGauge

AndyGauge Apr 23, 2018

Collaborator

There are a few ideas that seem dead in the water here. I like the rayon story in the cookbook and will close the tracking issue. If you want to see an additional recipe in the cookbook, open up a new issue.

Collaborator

AndyGauge commented Apr 23, 2018

There are a few ideas that seem dead in the water here. I like the rayon story in the cookbook and will close the tracking issue. If you want to see an additional recipe in the cookbook, open up a new issue.

@AndyGauge AndyGauge closed this Apr 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment