Benchmarks

[adscib]

This branch is not ready to merge yet!

Here's my proposal for how to set up benchmarks #7. We define a bunch of models in examples/ folder, each of which comes with appropriate performance metrics and a function that evaluates them all given a chain of samples from inference. On top of that there is a benchmarking script that runs the inference algorithm and measures execution time. We'll probably want it to supply an option to run the experiments multiple times and summarize the results. Finally, we can set up some notebooks to create custom plots for specific models.

Here's an example for how this would work on the Anglican Gaussian model. I know the plots are horrible, this is just to illustrate the setup. Any comments?

[yebai]

Looks good - I only have two organisational comments:

• Use a Benchmarking package, e.g. BenchmarkLite.
• It might be helpful to modify the travis.yml file to automatically execute the benchmarking script (maybe only for one simple model to keep testing time sensible)

[adscib]

Anglican examples are ready. I'll make some nice plots tomorrow. I also added a function localcopy that converts TArray into Array. I'm open to suggestions for a better name for it.

[yebai]

Cool - It seems that the purpose of localcopy is similar with Base.get (see tarray.jl). One minor difference is that Base.get does not deepcopy the data.

[adscib]

Why does Base.get take a task as an argument?

[yebai]

There are cases when Base.get is called by another task, e.g., in each sampler, we need to collect result after sampling is completed, then we need to call Base.get(particle_task, var).

[yebai]

I am noticing a strange behaviour with GitHub, whenever I create a breach from master, this PR get closed automatically and I have to re-open it. Sorry for the trouble!

[adscib]

I'm having some trouble with TArray. Could you take a look at the latest version of examples/anglican_crp.jl and see if I'm using something incorrectly (including the PolyaUrn definition in examples/utils.jl)? I'm getting "access to undefined reference" error.

[yebai]

I've fixed the bug in anglican_crp. The bug is due to the following line

  clusters = TArray{Distribution}(1)

will create an Array of length 1, but this only element is not initialised (null). The correct code should be

  clusters = TArray{Distribution}(0)

then clusters is of length 0, and will be expanded when new cluster is created.

[adscib]

KL plots for Anglican models are not looking good. Gaussian is OK, but SMC doesn't produce good results for HMM and CRP. At this point I have no idea what is the problem.

[yebai]

I've fixed the hmm model (see the commit). I couldn't find any problem with the crp model: it seems our sampler is converging to a consistent result. This makes me think that there might be a mis-match between our crp implementation and the anglican version. I will take a closer look tomorrow.