Compute confidence intervals in DistributedTreeDriver

[masterleinad]

Based on #78, this pull requests improves DistributedTreeDriver further. The idea is as follows:

• We first run the test a couple of time to have an estimate for mean and variance of the runtime for the individual sections (construction, knn, radius). (We take the maximum over all MPI processes in each iteration).
• These statistics are used to estimate for each section the number of total iterations needed to get a confidence interval of specified width.
• We run the maximum of these estimates number iterations (additionally) and output mean, variance and a new confidence interval at the end.

All this is based on CppCon 2015: Bryce Adelstein-Lelbach “Benchmarking C++ Code and Boost.Accumulators.

Sample output:

$ mpiexec -np 2 ./ArborX_DistributedTree.exe
ArborX version: 0.9 (dev)
ArborX hash   : 829b702

Running with arguments:
perform knn search      : true
perform radius search   : true
#points/MPI process     : 50000
#queries/MPI process    : 20000
size of shift           : 1
dimension               : 3


Sample lap 0:
contruction done
knn done
radius done

[...]

Sample lap 9:
contruction done
knn done
radius done

estimated 11 iterations

Total lap 10:
contruction done
knn done
radius done

[...]

Total lap 20:
contruction done
knn done
radius done
=========================================================================

TimeMonitor results over 2 processors

Timer Name   | mean         | variance     | confidence interval         
-------------------------------------------------------------------------
construction | 5.052878e-02 | 2.047170e-05 | [4.841836e-02, 5.263920e-02]
knn          | 1.172640e+00 | 2.565172e-02 | [1.097935e+00, 1.247345e+00]
radius       | 6.593428e-01 | 6.237161e-03 | [6.225057e-01, 6.961799e-01]
=========================================================================

[dalg24]

I need more time to review the math but here are a few questions/comments:

• I would probably want to merge Improve benchmark for distributed tree #78 first and rebase this onto master
• As much as I like the idea of computing confidence intervals and your using Boost.Accumulators I feel like the way TimeMonitor changes here is quite a stretch and it makes me question some of the design choices. I am more tempted to keep responsibilities separated: (i) a utility that measure time in a distributed context and (ii) a tool for statistics. (Please note that I do not advocate for the current design of TimeMonitor but I am even more skeptic about its evolution)

[masterleinad]

I have no idea why compiling with nvcc_wrapper fails...

[masterleinad]

So far I couldn't find parameters for boost::math::students_t::find_degrees_of_freedom such that the returned value would match the one computed here.

boost::math::students_t::find_degrees_of_freedom(relative_error_margin*current_mean, confidence/2, .9999999, current_stddev)

is just 1.3 times larger though.

[masterleinad]

I have no idea why compiling with nvcc_wrapper fails...

Just having a

 boost::accumulators::accumulator_set<double, ba::stats<ba::tag::count>>;

member variable gives

/opt/boost/include/boost/fusion/iterator/mpl/convert_iterator.hpp(57): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/iterator/mpl/convert_iterator.hpp(57): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/view/filter_view/filter_view_iterator.hpp(60): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/view/filter_view/filter_view_iterator.hpp(60): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/iterator/next.hpp(61): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/query/detail/find_if.hpp(208): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/iteration/detail/for_each.hpp(48): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/iteration/detail/for_each.hpp(36): error: identifier "" is undefined in device code
/opt/boost/include/boost/fusion/algorithm/iteration/detail/for_each.hpp(48): error: identifier "" is undefined in device code

when compiling with nvcc_wrapper.

[aprokop]

Anyone opposed to closing this? It was a nice attempt, but too complicated with little benefit in practice.

[masterleinad]

Yes, I think we won't do that.