opal/progress: added function to call a function as part of opal_progress

[hjelmn]

This commit add the opal_progress_run_once function. This function
creates a new event that is put on the opal_sync_event_base that runs
the requested function as part of opal_progress().

Signed-off-by: Nathan Hjelm hjelmn@lanl.gov

[hjelmn]

@jsquyres @rhc54 @bosilca This will replace the run_in_main stuff in the openib btl. I can move this function down there if there if this doesn't seem like good functionality to have up in opal/runtime.

[hjelmn]

If this goes into master it should be rolled up with the rest of the commits associated with #805.

[jsquyres]

I'd have to check, but I have a dim recollection that the run_in_main stuff was only because of the lack of thread safety with mpool... if that's correct, then there's no real need for this run_in_main stuff any more.

[hjelmn]

I think some of it had to do with the timing in the thread servicing rdmacm events.

[hjelmn]

I should add this also helps with the case when opal_using_threads() is 0. If there is an error the error needs to be invoked on the application thread and not the opal progress thread since it may cause an upcall into the pml (only bfo did this).

[jsquyres]

Are you sure?

The thread was just to ensure that RDMACM events could be processed in a timely fashion, but the progress thread couldn't actually do anything once the incoming connection request was fully processed (and therefore connected). So it threw the connection back up to the main thread.

If mpool is made thread safe, the need for "do something in the main thread as a proxy for my progress thread" functionality goes away. (I think / disclaimer: I haven't looked at the code)

[jsquyres]

@hjelmn and I chatted on the phone and figured it all out.

[hjelmn]

Sounds like this is not a function worth exposing. Will put it back down into the btl.

[bosilca]

👍 on the last @hjelmn comment. I don't think we want to encourage developers to start using single shot events on the main thread.