Skip to content

How_do_I_get_a_node_to_poll_its_children_in_parallel?

Gijs Molenaar edited this page Feb 12, 2014 · 1 revision

I have added multithreading to the kernel, so your trees can now be parallelized. To activate it, run the meqserver with the "-mt 2" (on lofar9/10) or "-mt 4" (on birch) option, which you can set in the "Connect..." dialog of the browser. (A space is required between -mt and the number.) The way it works is as follows:

  • any node with multiple children can be told to poll its children in parallel by creating it with mt_polling=true.
  • mt_polling is true by default for Solver, DataCollect and VisDataMux, and false for all other nodes. For most trees I can think of this is sufficient (there's no point in parallelizing beyond the condeq and subtract branches when we only have 2 or 4 CPUs), but in some special cases you may want to enable it on other nodes. E.g., Ronald was working on a tree that had a root Composer collecting data from a large number of UVBricks: in his case the Composer would certainly benefit from mt_polling=true.
  • By default the children are polled in the order they were attached, but you can set child_poll_order to a list of child names to change this. The children in the list are then polled in the specified order, after which any remaining (i.e. unlisted) children are polled. See MeqServer/test/matrix343.py for an example. In my observations it hasn't made much of a difference so far, probably due to caching helping out. In the 343 case multithreading can be almost 100% efficient (??). Running the source flux fit on 1 CPU takes 1:00~1:25 per iteration, while with 4 CPUs it's ~25 seconds. This of course is the best possible scenario -- intensive predict but a small number of unknowns. If you solve for a large number of unknowns, you don't win that much from parallel predicts because the solver becomes a bottleneck.

If something doesn't work right for you, remember that without the "-mt" option meqserver reverts to its old single-threaded behaviour, so please test your tree in both modes before reporting a problem.

Clone this wiki locally