Invalid out_of_bag_measures not caught leading to misleading output

[ablaom]

Unlike MLJBase.evaluate it seems MLJEnsembles design does not allow you to use deterministic measures for a probabilistic model. But rather than catching the offending measure, it silently applies it to give a wrong result:

using MLJ # or `MLJBase, MLJModels, MLJEnsembles` for minimal install

DecisionTreeClassifier = @iload DecisionTreeClassifier pkg=DecisionTree

atom = DecisionTreeClassifier()
model = EnsembleModel(
    atom;
    bagging_fraction=0.6,
    rng=123,
    out_of_bag_measure = [log_loss, accuracy],
)

X, y = @load_iris # a table and a vector

mach = machine(model, X, y) |> fit!
julia> report(mach).oob_measurements
2-element Vector{Float64}:
 2.0785173454390886
 0.0

The accuracy is not zero.

[ablaom]

I wonder if some of MLJBase/resampling.jl can be used or factored out to do this properly, with deterministic measures supported. Or at least, how complicated would that be?