rayon adaptive

[wagnerf42]

hi everyone,

you are all moving fast and i'm moving slowly but finally I wanted to present my work on rayon.

the idea has been to implement several techniques from our lab on top of rayon.

i've written a serie of blog posts here:
http://www-id.imag.fr/Laboratoire/Membres/Wagner_Frederic/rayon-adaptive.html

I don't present all my work but it is already a big chunk of it.
I would be very happy to discuss with everyone about it.

I hope you have some time during the holidays :-)

all feedback is welcome. thanks again for your wonderful library

[nikomatsakis]

Very cool! I am excited to read more deeply into those posts!

[wagnerf42]

hi niko,
I hope it will be interesting. there is more to come but i'm again a bit overwhelmed right now.

[cuviper]

Take note of #666, where a user is concerned about allocation overhead. In our last discussion, we were talking about Clone / Arc for your map closures -- both solutions may invoke the allocator in places where we don't now.

[wagnerf42]

thanks for the pointer. maybe I don't really get what's a closure. if I clone a closure isn't it just a pointer copy ? also, do you have an example of some closure which implements Fn but not Clone ? I still have no idea how to get one. finally, if someone gives a reference to a closure as argument will not the clone happen on the reference then ? Fred De: "Josh Stone" <notifications@github.com> À: "rayon-rs/rayon" <rayon@noreply.github.com> Cc: "frederic wagner" <frederic.wagner@imag.fr>, "Author" <author@noreply.github.com> Envoyé: Samedi 29 Juin 2019 03:27:25 Objet: Re: [rayon-rs/rayon] rayon adaptive (#616) Take note of [ #666 | #666 ] , where a user is concerned about allocation overhead. In our last discussion, we were talking about Clone / Arc for your map closures -- both solutions may invoke the allocator in places where we don't now. — You are receiving this because you authored the thread. Reply to this email directly, [ #616?email_source=notifications&email_token=ACN7RMYPDBLAYWSIJCOKOXDP42273A5CNFSM4GLX2CKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3OZBI#issuecomment-506915973 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/ACN7RM5JHZ7PSQXB4VR6PVLP42273ANCNFSM4GLX2CKA | mute the thread ] .

[cuviper]

maybe I don't really get what's a closure.

This post may help you:
https://krishnasannasi.github.io/rust/syntactic/sugar/2019/01/17/Closures-Magic-Functions.html

A closure is basically just a struct of the captured variables. If there are no captures at all, the closure will have zero size, not even a code pointer -- the code is statically known for a given closure type.

if I clone a closure isn't it just a pointer copy ?

For an immediate closure value, you're cloning the struct of captures -- which could just be references, but not necessarily. Cloning a reference &F where F: Fn is just a pointer copy though. Cloning a trait object like &dyn Fn would also be just a pointer copy -- actually a "fat" pointer pair of the struct pointer and a vtable pointer.

also, do you have an example of some closure which implements Fn but not Clone ?

By default, a Fn will only capture things by shared reference (&), and that trivially implements Clone, even Copy. However, this implies a lifetime on the closure for those references, which you sometimes want to avoid -- then you would use the move keyword on the closure. Then the closure will actually contain direct values owned in its struct, even though calling the closure still only has shared access through &self. These values may be expensive to clone (e.g. a large Vec as a lookup table), or might not implement Clone at all (e.g. a File).

finally, if someone gives a reference to a closure as argument will not the clone happen on the reference then ?

Yes, as &Fn also implements Fn, and then this is trivially both Clone and Copy. But note that your returned iterator like Map is now tied to the lifetime of that closure you referenced.

It's definitely a breaking change to add a Clone constraint where we currently accept Fn, and it would also make some move use cases impossible.

---

Adding Arc internally is a simpler solution, but introduces an allocation as noted.