Add c2st

[stefanradev93]

Adds classifier two-sample test (C2ST) metric to the bayesflow.diagnostics.metrics module. Example usage:

samples = ...
reference_samples = ...

# Using default arguments
c2st = classifier_two_sample_test(samples, reference_samples)

# Customizing arguments - will return full classification report + classifier and use a smaller validation data ratio
c2st_dict = classifier_two_sample_test(samples, reference_samples, return_metric_only=False, validation_split=0.2)

What we need to discuss is the conceptual separation of metrics, namely those intended to capture the performance of an ensemble of posteriors (e.g., SBC, RMSE) and depend on the prior or those intended to capture the quality of individual posteriors (e.g., MMD, C2ST). Should these two classes of metrics reside in different submodules? @paul-buerkner @vpratz @LarsKue

Note also, that I removed the internal cross-validation in favor of a simple train-test split. Since the C2ST will generally be computed on different source-target posterior samples, any epistemic uncertainty will average out. However, I could allow K-fold as well with some custom helpers, mainly because I wanted to avoid the explicit dependence on sklearn.

[codecov]

Codecov Report

Attention: Patch coverage is 82.55814% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
bayesflow/utils/classification/confusion_matrix.py	66.66%	8 Missing ⚠️
...ayesflow/utils/classification/calibration_curve.py	77.27%	5 Missing ⚠️
.../diagnostics/metrics/classifier_two_sample_test.py	93.33%	2 Missing ⚠️

Files with missing lines	Coverage Δ
bayesflow/diagnostics/metrics/__init__.py	100.00% <100.00%> (ø)
.../diagnostics/metrics/expected_calibration_error.py	100.00% <100.00%> (ø)
...yesflow/diagnostics/plots/calibration_histogram.py	90.90% <ø> (ø)
bayesflow/diagnostics/plots/mc_calibration.py	100.00% <100.00%> (ø)
bayesflow/diagnostics/plots/mc_confusion_matrix.py	89.74% <100.00%> (ø)
bayesflow/utils/__init__.py	100.00% <100.00%> (ø)
bayesflow/utils/classification/__init__.py	100.00% <100.00%> (ø)
.../diagnostics/metrics/classifier_two_sample_test.py	93.33% <93.33%> (ø)
...ayesflow/utils/classification/calibration_curve.py	77.27% <77.27%> (ø)
bayesflow/utils/classification/confusion_matrix.py	66.66% <66.66%> (ø)

... and 4 files with indirect coverage changes

[vpratz]

Thanks for the PR :)

Should these two classes of metrics reside in different submodules?

I like the idea, but lack good names that we could use for the respective submodules. Do you have an idea on how to distinguish them with two words that could serve as submodule names?

Note also, that I removed the internal cross-validation in favor of a simple train-test split.

As far as I can tell, using cross-validation gives us a more powerful test for the same number of samples, so I would opt against removing this in the default setting. How are other implementations handling this? Is there some default setup we might want to implement?
We use sklearn in expected_calibration_error and mc_confusion_matrix as well. Is the plan to refactor those as well, or are we going to keep the dependency anyway?

[paul-buerkner]

I think the PR looks good. Would you have time to add some tests too perhaps?

I would keep them in the same module for now even if their inputs are different. We clearly document the difference in input and I am not sure how we would name them if we went for two modules.

[stefanradev93]

@vpratz I removed all dependencies on sklearn across the library. @paul-buerkner Added tests for sensitivity of c2st.

[Copilot]

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

bayesflow/diagnostics/metrics/classifier_two_sample_test.py:21

• There is an inconsistency between the documented default value for mlp_widths (described as (256, 256)) and the actual default in the function signature ((64, 64)). Please update the docstring or the default value to be consistent.

mlp_widths: Sequence = (64, 64),