Allow Consumers to give a maximum split hint

[Plecra]

At this point, most (all?) implementations of {Indexed}ParallelIterator are implicitly tied to the rayon thread pool due to calling the current_num_threads function to determine the limit of the Splitter. This limits the usability of the traits in other threadpools, since they won't be optimized for the correct number of worker tasks.

To solve this, I think we could add a num_tasks method to the Consumer trait that could be used by the implementation of bridge*. By default, it could just return rayon::current_num_threads() for backwards compatibility.

Edit: If this doesn't make sense, it's possible I am entirely wrong in my understanding of the relationship between the traits. Part of the reason I think this would be helpful is because I expected rayon to provide an implementation of "number of worker threads" for the iterator implementations, and it wasn't clear to me that it was part of bridge*

[cuviper]

implicitly tied to the rayon thread pool

This limits the usability of the traits in other threadpools

You've piqued my interest -- you're using rayon traits with non-rayon thread pools?

[Plecra]

Haha, I hate to disappoint but I haven't got much further than an idea.

I'd only decided to finally learn how plumbing worked today, and while reading through I thought it might be possible to create a Consumer that constructed futures. This could possibly create a very ergonomic way of batching io operations within a single async task (in those situations where a stream can't be built and you're stuck with FuturesUnordered). Chances are there're a hundred other reasons the futures idea wouldn't work, but I opened this issue because it seemed against the spirit of generics to have this kind of assumption made by the implementations