Role of random phase in sinusoid example unclear

[matt-graham]

Raising as part of JOSS review openjournals/joss-reviews/issues/6372

In the model definition in the sinusoid example included in the documentation and JOSS paper

CalibrateEmulateSample.jl/docs/src/examples/sinusoid_example.md

Lines 66 to 69 in 6fcab82

	function model(amplitude, vert_shift)
	phi = 2 * pi * rand(rng)
	return amplitude * sin.(trange .+ phi) .+ vert_shift
	end

CalibrateEmulateSample.jl/docs/src/examples/sinusoid_example.md

Lines 110 to 114 in 6fcab82

	function G(theta)
	amplitude, vert_shift = theta
	sincurve = model(amplitude, vert_shift)
	return [maximum(sincurve) - minimum(sincurve), mean(sincurve)]
	end

it is not clear to me what the role of the random phase $\phi$ is and whether the model as defined fits into the framework assumed in the original Calibrate, Emulate, Sample (CES) paper and described in the JOSS paper and documentation.

The CES paper defines the generative model for the observations $y \in \mathbb{R}^d$ given parameters $\theta \in \mathbb{R}^p$ as

$$ y = \mathcal{G}(\theta) + \eta, \quad \eta \sim N(0, \Gamma_y) $$

with the forward map $\mathcal{G} : \mathbb{R}^p \to \mathbb{R}^d$ appearing to be assumed throughout to be deterministic.

The G / $\mathcal{G}$ function used in the example is in contrast stochastic due to the phase phi being randomly generated (using a global random number generator) within the function itself. Is there a reason for using a model which departs from the assumptions of the original paper as the main tutorial example?

I can see the value in being able to calibrate stochastic simulator models which internally make draws from a random number generator as this is typically a harder class of models to work with (and also very prevalent in practice), but its not clear to me if the CES framework straightforwardly extends to this case.

Relatedly there is a reference to approximate Bayesian computation (ABC) methods in the JOSS paper

In ABC, this approximation comes from bypassing the likelihood that is usually required in sampling methods, such as MCMC. Instead, the goal ABC is to replace the likelihood with a scalar-valued sampling objective that compares model and data. In CES, the approximation comes from learning the parameter-to-data map, then following this it calculates an explicit likelihood and uses exact sampling via MCMC

In the context of the above point about the use of a stochastic simulator model in the tutorial example I think the differences between CES and ABC are a bit muddied. ABC is typically applied to generative models for which we can simulate observable outputs given parameter values but cannot evaluate a density function for the joint distribution on observable outputs and parameters. ABC methods use a proxy for the density of the conditional distribution of the observables given parameters (likelihood) based on a distance measure between simulated observable outputs and observed data, typically using summary statistics of the observables rather than computing a distance on the full model outputs directly. Typically the assumption is that we can only compute a stochastic estimate of this proxy by sampling from the generative model.

The (random phase) sinusoid example model seems to fit in this description and so be a good candidate for applying ABC methods to. The joint distribution of $\mathcal{G}(\theta), \theta$ for this example is intractable (or at least non-trivial) to evaluate due to the random phase $\phi$. $\mathcal{G}$ effectively involves computing summary statistics of the full sinusoidal output, with the assumed Gaussian likelihood then equivalent to computing a Euclidean distance between the simulated and observed summary statistics, and as in ABC we only able to evaluate a stochastic estimate of this likelihood.

Given the additional complexity arising from the use of a stochastic simulator model, and it being unclear if / how the CES framework applies to such models, I would personally recommend just omitting the random phase aspect from the sinusoid example (by for example drawing a fixed random value for the phase phi outside the model function).

[odunbar]

The short and unsatisfying answer, is that the phase was included to make the example consistent with the related "simple example" in the EnsembleKalmanProcesses.jl package.

The original use was just a light comment about if you learn a shift/amplitude in an inverse problem, you may not need to know all details of the "model" f. In particular, the clever choice of G renders it actually deterministic (up-to numerical discretization error), even though f is a stochastic process, and the random phase shift has no affect on the inverse problem itself. However if instead we took G to be an observation of f pointwise, e.g. f(0) then I agree this would give a more interesting form of stochastic problem, with non-Gaussian noise and there CES would impose Gaussian likelihood constraints to approximate it.

With this in mind, CES should be applicable here, however there is ambiguity in exposition between the roles of f and G.

• If you would still like us to remove the shift \phi , then we could still do this.
• Alternatively In the sinusoid docs, we can highlight that our G is deterministic, despite f being stochastic and highlight that in setting up inverse problems, that one does not need to account for all model complexity to learn about parameters.

Let us know what you would prefer. I hope I interpreted your comment correctly

[matt-graham]

Sorry for the delay responding here. Thanks for clarification. I'd missed that f is almost determinisitic due to the choice of summary statistics here. I would say your second suggested option of adding some additional detail to the documentation would be fine here, as it was mainly that I was confused by the lack of comment on the random phase and what classes of model the CES framework is applicable to, but keeping thing consistent with an example in another package is I think a good reason to keep things as is.