Cookbook ideas for rayon

[KodrAus]

Come up with ideas for nice introductory examples of using the rayon crate, possibly in combination with other crates, that would be good to show in the Rust Cookbook. Please leave a comment here with your ideas! You don't necessarily have to write the example code yourself but PRs are always welcome!

rayon already has a suite of examples but they seem mostly focused on benchmarking.

Edit:

---

• Mutate the elements of an array in parallel
• Sort a vector in parallel
• Generate thumbnails for images in a directory in parallel
• Search unordered items in parallel
• Test if all/any items match predicate in parallel
• Parallel map reduce

[NoraCodes]

I wrote a Fizz Buzz example which might be appropriate: https://gist.github.com/LeoTindall/2bed40527d3920a73e9054fc80876b9c

It's (conveniently) fairly congruent with my Go example, which could be useful for those moving along the Go-to-Rust sidegrade path (https://gist.github.com/LeoTindall/433654da49c93c2da347bb7b5f7adc79).

[lilianmoraru]

I would like to see the rayon + features integration, but I guess this will turn into bigger, multi-line examples which might not be the most appropriate for the cookbook(don't know).

[budziq]

Hi @LeoTindall

Thanks for you interest! We would like to keep the cookbook focused on examples that can be considered:

1. working code snippet realizing some real life use-case
2. commonly needed building block that programmers might need

So FizzBuzz (parallel or not) might not be the best fit. Would you be able to think of an example that would fit into one of these categories?
You might consider using some of the ideas posted for threadpool

might not be the most appropriate for the cookbook(don't know).

@lilianmoraru - not necessarily. Do you have anything concrete in mind?

We might consider rewriting @ericho 's PR to use rayon (unfortunately dropping walkdir). Currently we have 2 super simple rayon examples and single relatively advanced threadpool one. I guess we should go with the one most ergonomic for the job .

[nocduro]

Rust/git noob here. I recently made a basic parallel image resizer with rayon here. I mostly used it to learn some more about rayon, so it is quite basic and only resizes images at the same directory level (not going into sub directories).

The hardest part for me was figuring out how to handle the errors, since it's not possible to use try! or ? on parallel code and as far as I saw none of the previous examples call a function that returns a Result. I'm still not sure if I'm doing the error handling correctly, so if anyone with more experience knows something better, let me know!

[budziq]

That is a very nice example @nocduro!
Are you willing to submit it as a PR? There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way! 👍

[nocduro]

Are you willing to submit it as a PR?

Sure! I've never made a pull request before, but I should be able to figure it out. I just read the CONTRIBUTING.md file as well.

There is nothing preventing parallel closures returning Result so the example can be simplified and pared down here and there. I'll gladly help you out along the way!

Awesome, thank you!

[danielpclark]

I think I have a good example idea for the Rayon Book example. I thought how I might be able to use Rayon for a regular iterator and it comes down to this.

First create a chunk builder method which will fill a vector of a certain quantity of work to work on. Then use Rayon's par_iter over each chunk. So any iterator can act as a Vec collection with Rayon if you chunk out the work.

For chunk size I went with 128 items or (in my new version) 32 times however many cores there are. I loop the chunks of work until I get my desired result. To see the code I have this implemented in here's an earlier version of my code that's simpler to look at: chunk_sequence and its use with par_iter. My project is a brute force password attempter for encrypted files with AES encryption. These code snippets shouldn't be hard to understand to base an example off of.

[budziq]

Thanks @danielpclark the technique is very interesting! Although we would like to focus on solving common real-life programming problems (brute decryption might not be all that common usecase). Would you be able to formulate the idea in terms of a programming task to solve?

[danielpclark]

@budziq How about don't call it brute forcing a password. Call it a game of guess the number. Otherwise I have no common use case.

[budziq]

@danielpclark guessing the number might not be an ideal use case but I'm sure there is a nice one somewhere in there but I cannot point my finger on it. If you come up with something that could be useful to a programmer trying to get things done see our inclusion policy draft please post it right away!

[danielpclark]

@budziq I've been putting a lot of thought into what you've asked for “common real-life programming problems”, “an ideal use case”, and “commonly needed building block that programmers might need”.

I've come to the conclusion that in general threading isn't common to many people and has more niche applications. The code example I've provided could easily by used for mathematical computations, or processing large collections of data through filters/tasks/jobs.

The problem I'm running in to is your question is a general programming question but the audience is too broad. If this were the R programming language I would have a better focus on who the examples were to be made for since they are a niche that focus on math, statistics, and learning algorithms. I don't know what the exact makeup of the Rust community is so I don't know which particular applications to design an example for.

The particular design for my example is as follows. We have a target variable which will hold the one answer the system is looking for in a Mutex. This doesn't have to be a one answer kind of thing, it can hold a Vector and separate threads can each call for a lock on it when they have something of relevance to put in it and build the data. Secondly my example has the loop to process input data in chunks and that is great for keeping track of performance and analytics as progress is being made. The par_iter is the no brainer part of Rayon which is super cool to use and just gets the job done. Beyond that this way of using Rayon has many potential applications and can be modified for any example as you see fit.

If you want me to produce an example I need to know the target audience. In my day to day life I don't process large amounts of anything and most of my work is sequential and trivial. So I don't commonly have much use of threads and I'm not in the habit of thinking of how to work in parallel. The other examples I might think of for web application doesn't work with pre-existing data but rather waits to spin up threads and that would be an entirely different example with a different implementation.

Without knowing the target audience I'm not sure any real “common, or ideal, use case” will arise.

[budziq]

@danielpclark Thanks for an explanation!

So I don't commonly have much use of threads

No worries and no pressure, neither do I. That is why there are so little concurrency examples to date ;)

I need to know the target audience.

It will be an extremely hard question to answer. Rust being general purpose language attracts users from a very broad spectrum (webdev, embedded, server, and many more, recently even some HPC folks started to take interest).

The best we can do here is to gather questions from the community in the form
"how am I to do XYZ with Rust and available crates" and most of the time we are acting as the source for these questions.

Rayon is especially hard in this matter as it magically just works. Most likely it would be conductive to some basic map-reduce scheme like the one written with std in RBE. Another approach would be to go over rayon's docs and search for inspiration while browsing its (quite large) feature-set.

[j-haj]

Any interest in parallel search using findy_any or any? The examples may be trivially small but nonetheless I think useful. If there is interest, I am happy to work on this.

[budziq]

@j-haj I'd say certainly yes on both accounts! If I were to showcase any id also show all. 👍

Just propose the working titles of both examples and we are ready to go. I'll create the issues.

---

Edit:
Already created the issue! Thanks for jogging me into this @j-haj and @danielpclark !

[AndyGauge]

There are a few ideas that seem dead in the water here. I like the rayon story in the cookbook and will close the tracking issue. If you want to see an additional recipe in the cookbook, open up a new issue.