Parallel processing of indexed for loops

[nicokoch]

Is there any way to add parallelism with rayons to loops that look like this:

for x in 1..g.width() - 1 {
        for y in 1..g.height() - 1 {
                ...processing....
        }
}

I do not even know if this would be possible in theory, but very helpful for some problems.

[nikomatsakis]

Something like this ought to do it:

(1..g.width() - 1)
    .into_par_iter()
    .flat_map(|i| (1..g.height() - 1).into_par_iter().map(|j| (i, j)))
    .map(|(i, j)| ... processing ...)

though I'd like to add a cross combinator, which would make it shorter.

[nicokoch]

I tried your example and simplified it a bit, but it gives me a lifetime error (I'm still really bad at resolving those). This is my code:

(1..1000usize)
    .into_par_iter()
    .flat_map(|x| (1..2000usize).into_par_iter().map(|y| (x, y)))
    .map(|(x, y)| {
        println!("{}, {}", x, y);
    });

And this is the error:

src/edges.rs:57:54: 57:64 error: `x` does not live long enough
src/edges.rs:57     .flat_map(|x| (1..2000usize).into_par_iter().map(|y| (x, y)))
                                                                     ^~~~~~~~~~
src/edges.rs:55:5: 57:66 note: reference must be valid for the method call at 55:4...
src/edges.rs:55     (1..1000usize)
src/edges.rs:56     .into_par_iter()
src/edges.rs:57     .flat_map(|x| (1..2000usize).into_par_iter().map(|y| (x, y)))
src/edges.rs:57:19: 57:65 note: ...but borrowed value is only valid for the scope of parameters for function at 57:18
src/edges.rs:57     .flat_map(|x| (1..2000usize).into_par_iter().map(|y| (x, y)))
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: aborting due to previous error
Could not compile `imageproc`.

To learn more, run the command again with --verbose.

Any idea how to resolve this?

[cuviper]

Try: .map(move |y| (x,y))
(this moves x into the closure, rather than accessing it by reference)

[nicokoch]

facepalm . That fixed it, thank you!

[nicokoch]

Another question just came up:

Assuming I had some more work to do in the outer loop, how would I do this with rayon? Example:

for x in 1..g.width() - 1 {
        for y in 1..g.height() - 1 {
                ...processing....
        }
    ...more_processing...
}

more_processing uses results produced by processing.
Thanks for any help in advance!

[cuviper]

Depends on what sort of processing you're hoping for. You could transcribe that more literally like:

(1..g.width() - 1).into_par_iter().for_each(|x| {
    (1 .. g.height() - 1).into_par_iter().for_each(|y| {
        /* processing */
    });
    /* more processing */
});

But remember those closures have to be Fn, not FnMut, so not all types of processing can be made parallel directly. Maybe instead of for_each, those should be nested .map.reduce ops.

[nikomatsakis]

As @cuviper said, something like:

(1..g.width() - 1).into_par_iter().for_each(|i| {
    let mut intermediate_result = vec![];
    (1..g.height() - 1).into_par_iter().map(|j| ...).collect_into(&mut intermediate_result);
    process(intermediate_result);
});

might be what you are looking for.

[nicokoch]

Exactly what I was looking for, thanks for your help guys!

[nikomatsakis]

Great :)