Iterate in parallel but with synchronisation

[Congyuwang]

I have this little project where I need concurrency for better performance. But I figured that it is not simple parallelisation.

Say you have tasks 0, 1, 2, 3, … and so on. Each task has two steps: 1, write to a cache; 2, read from a cache.

However, each task x, when reading, needs information written by task 0, 1, … x-1, and x.

For some time I thought this can only be done in single thread. But later I somehow figured a way to do such thing easily in parallel. The idea is:

1. just like rayon iter, you also use an iter, which runs in parallel.
2. However, unlike rayon, you get your output synced into sequential order.

Using such an iterator, you can solve the previous problem like this:
(0..end).iter(|x| {update_cache(x); x}).iter(|x| {read_cache(x)})

Because the iterator, although running in parallel, produces output in sequential order, when the second iter reads an ‘x’ from the first iterator, it is guaranteed that 0…x has been processed by the first iterator (I.e, write to cache).

Because I don’t find a crate to do this, I wrote a crate to do this myself:
https://github.com/Congyuwang/Synced-Parallel-Iterator

It has a relatively high overhead, but is extremely useful for my own project. I wonder if anyone has encountered similar need. If so, maybe someday this feature can be brought to rayon.

Simply call it ‘.into_par_iter_sync()’ would be great.

[adamreichold]

I think #858 and #210 discuss similar constructs.