Documentation updates. Make user interfaces slightly more readable by moving the @index block to the bottom of each page rather than the top. Fix links to example files

JoelTrent · JoelTrent · commit b66ff1b2ca4d · 2024-02-09T15:48:56.000+13:00
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@
 
 TO DO
 
-An implementation and exploration of the [likelihood-based profile-wise analysis workflow](https://doi.org/10.1371/journal.pcbi.1011515) from Matthew Simpson and Oliver Maclaren.
+This package is an implementation and exploration of the [likelihood-based profile-wise analysis workflow](https://doi.org/10.1371/journal.pcbi.1011515) from Matthew Simpson and Oliver Maclaren.
 
 The API is largely stable, but shouldn't be regarded as fixed until the first version is released; the package will be kept as v1.0.0-DEV until then. The main work remaining is in completing documentation. However, one of the bivariate boundary methods may be removed.
 
diff --git a/docs/src/examples/binomial_normal_approximation.md b/docs/src/examples/binomial_normal_approximation.md
@@ -1,6 +1,6 @@
 # Gaussian Approximation of a Binomial Distribution
 
-The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/-/tree/main/examples/binomial_normal_approximation.jl).
+The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/blob/main/examples/binomial_normal_approximation.jl).
 
 This model is taken from an unreleased paper by the authors of the workflow [simpsonprofilewise2023](@cite). The Binomial distribution is defined as:
 
diff --git a/docs/src/examples/logistic.md b/docs/src/examples/logistic.md
@@ -1,6 +1,6 @@
 # Logistic Model
 
-The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/-/tree/main/examples/logistic.jl).
+The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/blob/main/examples/logistic.jl).
 
 The logistic model with a normal data distribution [simpsonprofilewise2023](@cite) has the following differential equation for the population density ``C(t)\geq0``:
 ```math
diff --git a/docs/src/examples/lotka-volterra.md b/docs/src/examples/lotka-volterra.md
@@ -1,6 +1,6 @@
 # Lotka-Volterra Model
 
-The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/-/tree/main/examples/lotka-volterra.jl).
+The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/blob/main/examples/lotka-volterra.jl).
 
 The Lotka-Volterra model with a normal data distribution [simpsonprofilewise2023](@cite) has the following differential equations for the population size of the prey species ``x(t)`` and predator species ``y(t)``:
 
diff --git a/docs/src/examples/two-species_logistic.md b/docs/src/examples/two-species_logistic.md
@@ -1,6 +1,6 @@
 # Two-Species Logistic Model
 
-The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/-/tree/main/examples/two-species_logistic.jl).
+The code included in this example is compiled into a single file [here](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl/blob/main/examples/two-species_logistic.jl).
 
 The two-species logistic model with a Gaussian data distribution [simpsonprofile2023](@cite) has the following differential equations for the population densities of the two species ``C_1(t)\geq0`` and ``C_2(t)\geq0`` :
 ```math
diff --git a/docs/src/index.md b/docs/src/index.md
@@ -6,3 +6,51 @@ CurrentModule = LikelihoodBasedProfileWiseAnalysis
 
 Documentation for [LikelihoodBasedProfileWiseAnalysis](https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl).
 
+This package is an implementation and exploration of the [likelihood-based Profile-Wise Analysis (PWA) workflow](https://doi.org/10.1371/journal.pcbi.1011515) from Matthew Simpson and Oliver Maclaren [simpsonprofilewise2023](@cite). It provides methods for:
+
+- Maximum Likelihood Estimation.
+- Calculation of the observed Fisher information matrix (FIM) [pawitanall2001](@cite) and associated approximation of the log-likelihood function.
+- Parameter identifiability analysis.
+- Parameter confidence intervals.
+- Evaluating univariate profiles.
+- Evaluating the boundaries of bivariate profiles and sampling points within these boundaries.
+- Rejection sampling of full parameter vector confidence sets and profiles.
+- Simultaneous prediction of model solutions/trajectories using approximate profile-wise confidence trajectory sets.
+- Simultaneous prediction of population reference sets using approximate profile-wise reference tolerance sets.
+
+Additionally, to assist with evaluating the frequentist coverage properties of intervals and sets within the PWA workflow on new models it provides methods for the coverage testing of:
+- Parameter confidence intervals.
+- Bivariate confidence profiles.
+- Profile-wise confidence trajectory sets.
+- Profile-wise reference tolerance sets.
+
+To understand the background of the workflow and how it can be used see [Background](@ref). For implementation examples see the examples section, such as on a [Logistic Model](@ref). To better understand how to interact with the user interface and in particular the [`LikelihoodModel`](@ref), which holds all the information on computed profiles and predictions, check out the user interface starting with [Initialisation](@ref).
+
+The API is largely stable, but shouldn't be regarded as fixed until the first version is released; the package will be kept as v1.0.0-DEV until then. The main work remaining is in completing documentation. However, one of the bivariate boundary methods may be removed.
+
+A package developed to fulfil the requirements of a Masters of Engineering at The University of Auckland by Joel Trent between March 2023 and February 2024. 
+
+Supervised by Oliver Maclaren, Ruanui Nicholson and Matthew Simpson.
+
+## Getting Started: Installation
+
+To install the package, use the following command inside the Julia REPL. In the future this will be able to be installed directly rather than via the url.
+
+```julia
+using Pkg
+Pkg.add(url="https://github.com/JoelTrent/LikelihoodBasedProfileWiseAnalysis.jl")
+```
+
+To load the package, use the command:
+
+```julia
+using LikelihoodBasedProfileWiseAnalysis
+```
+
+## Alternatives for Likelihood-Based Uncertainty Quantification
+
+If you are solely interested in parameter identifiability analysis and computing parameter confidence intervals we recommend [LikelihoodProfiler](https://insysbio.github.io/LikelihoodProfiler.jl/stable/), which is generally more stable and faster than the implementation in this package on the models we've tested.
+
+[ProfileLikelihood](https://danielvandh.github.io/ProfileLikelihood.jl/stable/) is another excellent package which implements the PWA workflow from [simpsonprofilewise2023](@cite) - it has a different interface and its own set of heuristics for computing profiles. 
+
+[InformationGeometry](https://rafaelarutjunjan.github.io/InformationGeometry.jl/stable/) can compute the exact confidence regions/boundaries of models using differential geometry. It would be an interesting approach which could potentially evaluate confidence set boundaries more efficiently than the heuristics implemented in this package. Resultantly, its use within the PWA workflow may be worth investigating.
diff --git a/docs/src/quick_start.md b/docs/src/quick_start.md
diff --git a/docs/src/refs.bib b/docs/src/refs.bib
@@ -1,3 +1,43 @@
+@book{coleparameter2020,
+	location = {New York},
+	title = {Parameter Redundancy and Identifiability},
+	isbn = {978-1-315-12000-3},
+	pagetotal = {272},
+	publisher = {Chapman and Hall/{CRC}},
+	author = {Cole, Diana},
+	year = {2020},
+	url = {https://doi.org/10.1201/9781315120003},
+}
+
+@article{murphyimplementing2023,
+	title = {Implementing measurement error models in a likelihood-based framework for estimation, identifiability analysis, and prediction in the life sciences},
+	author = {Murphy, Ryan J. and Maclaren, Oliver J. and Simpson, Matthew J.},
+	doi = {10.48550/arXiv.2307.01539},
+	year = {2023},
+	journal = {arXiv:2307.01539}
+}
+
+@book{pawitanall2001,
+	title = {In All Likelihood: Statistical Modelling and Inference Using Likelihood},
+	isbn = {978-0-19-850765-9},
+	shorttitle = {In All Likelihood},
+	pagetotal = {528},
+	publisher = {Oxford University Press},
+	author = {Pawitan, Yudi},
+	year = {2001},
+}
+
+@article{rauestructural2009,
+    title = "{Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood}",
+    volume = {25},
+    url = {https://doi.org/10.1093/bioinformatics/btp358},
+    pages = {1923-1929},
+    number = {15},
+    journal = {Bioinformatics},
+    author = {Raue, A. and Kreutz, C. and Maiwald, T. and Bachmann, J. and Schilling, M. and Klingmüller, U. and Timmer, J.},
+    year = {2009},
+}
+
 @article{simpsonprofilewise2023,
 	title = {Profile-Wise Analysis: A profile likelihood-based workflow for identifiability analysis, estimation, and prediction with mechanistic mathematical models},
 	volume = {19},
@@ -18,36 +58,15 @@ @article{simpsonprofile2023
 	year = {2023},
 }
 
-@article{rauestructural2009,
-    title = "{Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood}",
-    volume = {25},
-    url = {https://doi.org/10.1093/bioinformatics/btp358},
-    pages = {1923-1929},
-    number = {15},
-    journal = {Bioinformatics},
-    author = {Raue, A. and Kreutz, C. and Maiwald, T. and Bachmann, J. and Schilling, M. and Klingmüller, U. and Timmer, J.},
-    year = {2009},
-}
-
-@book{pawitanall2001,
-	title = {In All Likelihood: Statistical Modelling and Inference Using Likelihood},
-	isbn = {978-0-19-850765-9},
-	shorttitle = {In All Likelihood},
-	pagetotal = {528},
-	publisher = {Oxford University Press},
-	author = {Pawitan, Yudi},
-	year = {2001},
-}
-
-@book{coleparameter2020,
-	location = {New York},
-	title = {Parameter Redundancy and Identifiability},
-	isbn = {978-1-315-12000-3},
-	pagetotal = {272},
-	publisher = {Chapman and Hall/{CRC}},
-	author = {Cole, Diana},
-	year = {2020},
-	url = {https://doi.org/10.1201/9781315120003},
+@article{simpsonreliable2022,
+	title = {Reliable and efficient parameter estimation using approximate continuum limit descriptions of stochastic models},
+	volume = {549},
+	issn = {0022-5193},
+	doi = {10.1016/j.jtbi.2022.111201},
+	pages = {111201},
+	journal = {Journal of Theoretical Biology},
+	author = {Simpson, Matthew J. and Baker, Ruth E. and Buenzli, Pascal R. and Nicholson, Ruanui and Maclaren, Oliver J.},
+	year = {2022}
 }
 
 @article{viallefontparameter1998,
diff --git a/docs/src/user_interface/coverage/univariate_intervals.md b/docs/src/user_interface/coverage/univariate_intervals.md
@@ -8,4 +8,4 @@ Usage on several models can be seen in the examples section, such as for the [Lo
 
 ```@docs
 check_univariate_parameter_coverage
-```
+```
diff --git a/docs/src/user_interface/initialisation.md b/docs/src/user_interface/initialisation.md
@@ -1,9 +1,5 @@
 # Initialisation
 
-```@index
-Pages = ["initialisation.md"]
-```
-
 ## Model Initialisation
 
 To initialise a model for profile likelihood evaluation we use [`initialise_LikelihoodModel`](@ref) which returns a struct of type [`LikelihoodModel`](@ref). This struct contains all the information we require for the PWA workflow and will also contain computed profiles and profile-wise predictions.
@@ -59,4 +55,10 @@ set_OptimizationSettings!
 OptimizationSettings
 optimise
 optimise_unbounded
+```
+
+## Index
+
+```@index
+Pages = ["initialisation.md"]
 ```
diff --git a/docs/src/user_interface/plots.md b/docs/src/user_interface/plots.md
@@ -1,9 +1,5 @@
 # Plots
 
-```@index
-Pages = ["plots.md"]
-```
-
 We provide functions for plotting univariate profiles, bivariate profiles and predictions of the model trajectory and ``(1-\delta)`` population reference set from evaluated parameter confidence sets (i.e. profiles).
 
 ## Univariate Profiles
@@ -28,4 +24,10 @@ plot_predictions_individual
 plot_predictions_union
 plot_realisations_individual
 plot_realisations_union
+```
+
+## Index
+
+```@index
+Pages = ["plots.md"]
 ```
diff --git a/docs/src/user_interface/predictions.md b/docs/src/user_interface/predictions.md
@@ -1,9 +1,5 @@
 # Predictions
 
-```@index
-Pages = ["predictions.md"]
-```
-
 ## Adding a Prediction Function
 
 In the event that a prediction function has not been added to the [`LikelihoodModel`](@ref) struct yet, we can add one using [`add_prediction_function!`](@ref). This prediction function is for model solutions/trajectory and not the additional error we account for when predicting realisations.
@@ -53,4 +49,10 @@ Predictions are stored in a [`PredictionStruct`](@ref) which will also contain a
 AbstractPredictionStruct
 PredictionStruct
 PredictionRealisationsStruct
+```
+
+## Index
+
+```@index
+Pages = ["predictions.md"]
 ```
diff --git a/docs/src/user_interface/profiles_and_samples/bivariate.md b/docs/src/user_interface/profiles_and_samples/bivariate.md
@@ -1,8 +1,6 @@
 # Bivariate Profiles
 
-```@index
-Pages = ["bivariate.md"]
-```
+The key function for evaluating bivariate profile boundaries is [`bivariate_confidenceprofiles!`](@ref). The evaluated bivariate profile(s) will be contained within a [`BivariateConfidenceStruct`](@ref) that is stored in the [`LikelihoodModel`](@ref).
 
 ```@docs
 bivariate_confidenceprofiles!
@@ -40,9 +38,15 @@ MPPHullMethod
 
 ## Merging Boundaries From Multiple Methods
 
-To improve the performance of internal point sampling, it may be worth finding bivariate boundaries using a combination of methods, where one method has more guaranteed boundary coverage and the other gives a more random search of interest parameter space, such as combining [`IterativeBoundaryMethod`](@ref) with [`SimultaneousMethod`](@ref) into a [`CombinedBivariateMethod`](@ref). 
+To improve the performance of internal point sampling, it may be worth finding bivariate boundaries using a combination of methods, where one method has more guaranteed boundary coverage and the other gives a more random search of interest parameter space. For example, combining [`IterativeBoundaryMethod`](@ref) and [`SimultaneousMethod`](@ref) into a [`CombinedBivariateMethod`](@ref). 
 
 ```@docs
 CombinedBivariateMethod
 combine_bivariate_boundaries!
+```
+
+## Index
+
+```@index
+Pages = ["bivariate.md"]
 ```
diff --git a/docs/src/user_interface/profiles_and_samples/dimensional.md b/docs/src/user_interface/profiles_and_samples/dimensional.md
@@ -1,13 +1,11 @@
 # Dimensional Samples
 
-```@index
-Pages = ["dimensional.md"]
-```
-
 We can generate samples within the log-likelihood boundary at any dimension of model interest parameters. Nuisance parameters will be set to the values that maximise the log-likelihood function, found using an optimisation scheme. Samples are only implemented for the true log-likelihood function (the [`LogLikelihood`](@ref) profile type).
 
 ## Sample Types
 
+Three methods are implemented for rejection sampling of parameter confidence sets within the supplied parameter bounds. These will be inefficient if the bounds are not well-specified (are not close to the boundary of the desired parameter confidence set), and as both the model parameter dimension increases and the interest parameter dimension of the profile increases. By default these are the bounds contained with [`CoreLikelihoodModel`](@ref), however, seperate bounds can be supplied for the purposes of sampling. Uniform grid sampling, uniform random sampling and sampling from a random Latin Hypercube scheme (the default) are supported.
+
 ```@docs
 AbstractSampleType
 UniformGridSamples
@@ -17,14 +15,24 @@ LatinHypercubeSamples
 
 ## Full Likelihood Sampling
 
+To sample a confidence set for the full parameter vector, a full parameter confidence set, we use [`full_likelihood_sample!`](@ref). The sampled confidence set will be contained within a [`SampledConfidenceStruct`](@ref) that is stored in the [`LikelihoodModel`](@ref).
+
 ```@docs
 full_likelihood_sample!
 ```
 
 ## Dimensional Likelihood Sampling
 
-Note: dimensional likelihood samples can be 'full' likelihood samples as well.
+Similarly, to sample a confidence set for an interest subset of the parameter vector, a 'dimensional profile', we use [`dimensional_likelihood_samples!`](@ref). The sampled confidence set(s) will be contained within a [`SampledConfidenceStruct`](@ref) that is stored in the [`LikelihoodModel`](@ref).
+
+Note: dimensional likelihood samples can be 'full' likelihood samples as well. These will be computed before any other dimensional samples.
 
 ```@docs
 dimensional_likelihood_samples!
+```
+
+## Index
+
+```@index
+Pages = ["dimensional.md"]
 ```
diff --git a/docs/src/user_interface/profiles_and_samples/profile_structs.md b/docs/src/user_interface/profiles_and_samples/profile_structs.md
@@ -1,11 +1,9 @@
 # Structs and Profile Types
 
-```@index
-Pages = ["profile_structs.md"]
-```
-
 ## Structs
 
+The following structs are used to contain information on computed univariate, bivariate and sampled dimensional profiles. After evaluation they are stored in the [`LikelihoodModel`](@ref).
+
 ```@docs
 PointsAndLogLikelihood
 AbstractConfidenceStruct
@@ -24,4 +22,10 @@ AbstractEllipseProfileType
 LogLikelihood
 EllipseApprox
 EllipseApproxAnalytical
+```
+
+## Index
+
+```@index
+Pages = ["profile_structs.md"]
 ```
diff --git a/docs/src/user_interface/profiles_and_samples/univariate.md b/docs/src/user_interface/profiles_and_samples/univariate.md
@@ -1,12 +1,16 @@
 # Univariate Profiles
 
-```@index
-Pages = ["univariate.md"]
-```
+The key function for evaluating univariate profiles (the parameter confidence interval and corresponding points along the profile as desired) is [`univariate_confidenceintervals!`](@ref).  The evaluated univariate profile(s) will be contained within a [`UnivariateConfidenceStruct`](@ref) that is stored in the [`LikelihoodModel`](@ref).
 
 ```@docs
 univariate_confidenceintervals!
 get_points_in_intervals!
 get_uni_confidence_interval
 get_uni_confidence_interval_points
+```
+
+## Index
+
+```@index
+Pages = ["univariate.md"]
 ```
diff --git a/docs/src/workflow_background.md b/docs/src/workflow_background.md
@@ -0,0 +1,8 @@
+# Workflow Background
+
+Here we cover the background / general idea of the PWA workflow as seen in [simpsonprofilewise2023](@cite). We also introduce the idea of reference tolerance sets for predictions of realisations/observations within the workflow as introduced in my [Masters thesis](https://github.com/JoelTrent/UoA_MastersWorking).
+
+This is a summary of the content in my [Masters thesis](https://github.com/JoelTrent/UoA_MastersWorking).
+
+This summary uses models of the form 'deterministic mathematical model + error model' as in [simpsonprofilewise2023](@cite) and [murphyimplementing2023](@cite), but it can also be used with stochastic models [simpsonreliable2022](@cite).
+
diff --git a/src/model_initialiser.jl b/src/model_initialiser.jl

-Original file line number
+Diff line change
 - `dim_row_preallocation_size`: number of rows of `dim_samples_df` to preallocate. Default is `missing` (a single row).
 - `find_zero_atol`: a `Real` number greater than zero for the absolute tolerance of the log-likelihood function value from the target value to be used when searching for confidence intervals/boundaries. Default is `0.001`.
 - `show_progress`: Whether to show the progress of profiling and predictions.
++
 +!!! note "Array initialisation within a log-likelihood function"
 +    If you initialise an array within the provided log-likelihood function, e.g. using `zeros`, then for automatic differentiation methods to work you need to also initialise the type of the array to be based on the type of the input values of θ. Otherwise, zeros will by default create an array with element types `Float64` which will likely return errors. For example, use:
 +    ```julia
 +    my_new_array = zeros(eltype(θ), dimensions)
 +    ```
 +    where `eltype` passes the element type of the θ vector into the `zeros` function.
 """
 function initialise_LikelihoodModel(loglikefunction::Function,
     predictfunction::Union{Function, Missing},