[GENERAL SUPPORT]: How is standard error incorporated into the Gaussian Process?

[claysmyth]

Question

Dear Ax Community,

Thank you for this wonderful tool. I am currently using Ax to optimize field experiments for invasive neuromodulatory procedures. I'm writing to ask how the standard errors are incorporated into the Gaussian Process surrogate model.

I am interested in incorporating variable observation noise around my measurements. My understanding of how this heteroskedastic noise is incorporated comes from Garnett 2023 - Bayesian Optimization:

Where:

I noticed that when calling ax_client.complete_trial() the user can input a standard error around each observation. I want to confirm that the SEM around each observation corresponds the diagonal elements of N matrix in the second screenshot above?

Additionally, how is the baseline noise around measurements typically inferred? I want to use the noise around measurements as a means of implementing 'confidence' in a specific measurement, and would love guidance on choosing the right scale for including this metadata.

Thank you for your time!

Best,
Clay Smyth

Please provide any relevant code snippet if applicable.

Code of Conduct

• I agree to follow this Ax's Code of Conduct

[Balandat]

I want to confirm that the SEM around each observation corresponds the diagonal elements of N matrix in the second screenshot above?

Up to a square, yes - SEM denotes the standard error of the measurement*, but N contains the variances.

*SEM is really a misnomer here, it should be standard error. The reason it's called SEM is for historical in that Ax use cases initially focused on parameter tuning for online experiments, where the observed outcome is the mean of some population (and the associated standard error is the standard error of that mean, hence sem).

This asterisk above also partially answers your other question - in some applications we actually do observe (at least an estimate) of the variance of the observation. In other cases you may want to run some metastudy and evaluate some measurements repeatedly to estimate the variance from those repeated observations (this will be most scalable if your noise is homoskedastic).

I want to use the noise around measurements as a means of implementing 'confidence' in a specific measurement, and would love guidance on choosing the right scale for including this metadata

Do you have the variance of these measurements / can you estimate it? Or is this more about trying to somehow incorporate some domain knowledge / user prior without having a well-defined notion of measurement noise? This paper is quite relevant to your question, I recommend you take a look at it and the references therein to get an idea of what people have done in these cases: https://proceedings.neurips.cc/paper_files/paper/2024/hash/7124771cf8c9b0f6e9c7fec0a66c5866-Abstract-Conference.html

[claysmyth]

@Balandat Thank you for the thoughtful response. Your paper is very interesting, we will take note of it in our pipelines. I am still parsing the paper and it's citations, which have been helpful.

To answer your question, we are recording a statistic from a time series recording of brain activity, and using this statistic as our reward function over which we are optimizing. Unfortunately, the length of these recordings (i.e. number of samples) can vary dramatically. We would like to increase measurement variance for statistics gathered from short recordings, as we are 'less certain' in these statistics because they pooled from recordings with less samples.

We think a good candidate for implementing this measurement variance would be calculating the standard error for an observation of brain activity as:

1 / sqrt(t)

Where t = brain recording time in minutes. Shorter recordings would correspond to larger standard error.

However, these recordings are typically hundreds of minutes long. Would our standard error be 'washed out' by the standard deviation of our kernel function if we use Ax out-of-the-box?

[Balandat]

Interesting application. Too bad for you you're not working on my brain, things would be a lot easier there (no activity to speak of).

We think a good candidate for implementing this measurement variance would be calculating the standard error for an observation of brain activity as 1 / sqrt(t) where t = brain recording time in minutes.

This sounds quite heuristic. If you have a statistic defined for the time series, could you not express / estimate the variance in the statistic in a more principled fashion (e.g. by viewing the end of the time series as some random stopping time)? You could potentially simulate this by randomly stopping longer curves based on the empirical distribution of stopping times, recompute the respective statistics on the censored curve, and then use the variance of that somehow... - I'm speculating here, probably good to talk to a proper statistician :)

Would our standard error be 'washed out' by the standard deviation of our kernel function if we use Ax out-of-the-box?

It's sort of hard to say that upfront. It really depends on how much signal-to-noise ratio there is in your data.

[claysmyth]

@Balandat I'm sure your brain activity is off-the-charts!

Modeling the time series length as a random variable does seem to address the problem in an interesting way, and is potentially more principled. Thanks a ton for your input, I appreciate your thoughtful responses!

If we come to a nice conclusion, I will share here.

Cheers.

[lena-kashtelyan]

Hi @claysmyth , looks like this was resolved –– it'll be super interesting to hear your results! Please reopen this issue when you share them, we'd be very interested to hear them : ) I'm closing the issue for now to make sure that we see the new issues and keep our total issues in check, but please do reopen this if you comment. We might not see a new comment on a closed issue and we wouldn't want to miss this one!