Updated first examples to show model-free case.

[MichaelClerx]

After discussion today, seems like we should start the first examples by showing you need an error/logpdf, then move on to show how PINTS lets you define one for time series

[MichaelClerx]

@ben18785 @chonlei @martinjrobins do you think this makes sense? It was a little unclear that you could use PINTS for any old function

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (89fd597) to head (98f27a4).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #1520   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           97        97           
  Lines         9695      9695           
=========================================
  Hits          9695      9695           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[k-shep]

On a related topic, I think the readme could also be updated to make it clear that PINTS can also be used for simpler problems. For instance, currently it starts by saying

'PINTS (Probabilistic Inference on Noisy Time-Series) is a framework for optimisation and Bayesian inference on ODE models of noisy time-series, such as arise in electrochemistry and cardiac electrophysiology.'

I think it would be good to make it clear that it can also be used with other models that aren't ODEs.

[k-shep]

I think this parabola problem is a really good simple first example. (I did actually think you couldn't use PINTS for 1D optimisation though - I'm pretty sure I encountered an error when I tried to use it for a 1D problem a while ago, and I thought someone else had told me that they'd also experienced this.)

My thoughts

Parabola example:

1. I'd use 'PINTS' rather than 'Pints' throughout
2. I don't think I was really aware of the ErrorMeasure class. For me, maybe some explanation of when you'd use it vs the logPDF might be helpful. I'm guessing you just use it when you don't have a logPDF?
3. I think for beginners it might be helpful to have a bit more of an explanation of the output of PINTS and what it is doing - e.g. it's running the Nelder-Mead optimisation algorithm and this algorithm is evaluating the function at different points. I guess briefly explaining what is shown in each of the columns of the output might be helpful (if this isn't done elsewhere).
4. I think plotting the parabola and the minimum that has been found might be useful (or at least printing found_parameters, found_value).
5. I found the sentence 'Anything that can be made to fit the ErrorMeasure interface can be optimised this way, including error measures that run simulations, make calls to other software packages, or even interface with other languages.' a bit confusing. I'd maybe change 'error measures' to 'classes' or 'functions'.

Fitting time-series models

1. I was wondering actually if this notebook should be split in 2 - so at the bottom of parabola example there is just a hyperlink to this example. Perhaps there could be different first examples depending on what someone is looking to do?
2. I also found the following sentence confusing: 'As with the error function above, there are only two main methods: One to tell the optimiser how many parameters the are, and one to run the simulations.' I guess we evaluate the function, but for me this is different from running a simulation. Maybe something like '...one to evaluate the model or run a simulation'?

[k-shep]

I think a simple example where a logPDF is maximised, or just explaining that this can be done in the same way as for the parabola example, might also be helpful. Alternatively, it might be useful to add a separate notebook with this, if there's not already one. For me starting out I think a simple example where the following is done would have been useful:

1. How to define a log-likelihood
2. How to find the MLE
3. How to define a prior and find the posterior

I'd be happy to try and make a notebook doing the above, if that would be useful.

[k-shep]

I also really like this new example. I think it would be worth adding at the very start that, in the first simple example, we're just drawing samples from a distribution, and we're not performing inference until we get to the main one.

Other things I spotted

1. This has a strange looking scrolling bit for me for the first bit of text (see the RHS of the picture below)
   

2. I think Markov, Gaussian should be capitalised

3. At the very end of the notebook, there's a small typo in 'these'. It is written as thes.

[MichaelClerx]

Thanks @k-shep ! I'll deal with the smaller points shortly.

With regards to the main introduction I think we should discuss in the meeting with the other @pints-team/devs ! We'll need to find a balance between the original USP (inference for time series) and wider applications - or decide to change PINTS' focus.

[MichaelClerx]

(I did actually think you couldn't use PINTS for 1D optimisation though - I'm pretty sure I encountered an error when I tried to use it for a 1D problem a while ago, and I thought someone else had told me that they'd also experienced this.)

I'm guessing you did what I did initially, and used a scalar parameter instead of a length-1 vector?

[k-shep]

(I did actually think you couldn't use PINTS for 1D optimisation though - I'm pretty sure I encountered an error when I tried to use it for a 1D problem a while ago, and I thought someone else had told me that they'd also experienced this.)

I'm guessing you did what I did initially, and used a scalar parameter instead of a length-1 vector?

I can't fully remember, but from looking back at my code I think you're probably correct. I do remember seeing an error raised by PINTS which said that it didn't support the 1D case. I think maybe @I-Bouros experienced something similar.

[I-Bouros]

(I did actually think you couldn't use PINTS for 1D optimisation though - I'm pretty sure I encountered an error when I tried to use it for a 1D problem a while ago, and I thought someone else had told me that they'd also experienced this.)

I'm guessing you did what I did initially, and used a scalar parameter instead of a length-1 vector?

I can't fully remember, but from looking back at my code I think you're probably correct. I do remember seeing an error raised by PINTS which said that it didn't support the 1D case. I think maybe @I-Bouros experienced something similar.

That's true from my experience PINTS does not work with optimising or sampling single parameters.

[MichaelClerx]

OK Can you dig out a specific example? The notebooks in this PR show both a 1d optimisation and a 1d sampling problem, so it's definitely not impossible in general :D

[I-Bouros]

OK Can you dig out a specific example? The notebooks in this PR show both a 1d optimisation and a 1d sampling problem, so it's definitely not impossible in general :D

I can describe the phenomenon, as it appears in my own work: I am working with an SEIR model for which I am trying to infer one parameter beta. I use my usual approach in which I write up a pints.LogPDF, pints.LogPrior and a pints.LogPosterior objects, and then when I try to run it (with 3 chains, so is not that issue), I get this error message:

[MichaelClerx]

Oh, that's interesting! Seems to be coming directly from CMA-ES. We should add a notice to the CMA-ES optimiser to say it doesn't support 1-d optimisations. Alternatively could try and catch this error and replace it with a more informative one

[MichaelClerx]

• The readme could be updated to make it clear that PINTS can also be used for simpler problems (than ODEs).
• Pints --> PINTS
• Explain ErrorMeasure vs LogPDF
• More explanation of the output of PINTS and what it is doing
• Optimisation: Printing found_parameters, found_value
• Confusing sentence 'Anything that can be made to fit the ErrorMeasure interface...'
• Confusing sentece: 'two main methods: One to tell the optimiser how many parameters the are, and one to run the simulations'
• Optimisation: Show how a LogPDF can be used too
• Capitalise Markov, Gaussian
• Sampling: "Thes" --> "These"
• Note about 1D Updated docs for optimiser and CMA-ES. Added proper exception for CMAES not supporting 1-d. #1698