You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A metric's model attribute, a ModelInterface, gives an
estimator of your own what Empulse's models use of a loss: its parameters, decision costs,
decision threshold, loss, training objectives and cost scale. See User Guide.
Direction, which a custom MetricStrategy
passes to its constructor and every metric reports as direction, is exported from metrics. It could only be imported from a private module.
[constrain>](https://empulse.readthedocs.io/en/stable/reference/generated/CostMatrix.constrain <empulse.metrics.CostMatrix.html#CostMatrix.constrain <empulse.metrics.CostMatrix.constrain>) and
[constrain>](https://empulse.readthedocs.io/en/stable/reference/generated/MixtureMetric.constrain <empulse.metrics.MixtureMetric.html#MixtureMetric.constrain <empulse.metrics.MixtureMetric.constrain>) take a condition relating
several parameters, written as a comparison such as 'clv > incentive_cost' or as a sympy
relation, which may name aliases. Unlike a lambda, which cannot be pickled, it keeps the metric and
the models fitted with it picklable, for instance by dump. A condition on a name that
is no symbol or alias raises when the metric is built.
A MixtureMetric of Savings components
decides and trains on the weighted sum of their savings rather than of their costs, so the
component with the largest costs no longer decides alone. This applies to optimal_threshold, optimal_rate, the training objectives, the cost-sensitive trees, CSThresholdClassifier and CostSensitiveSampler.
Because the weights now depend on the labels, such a mixture can no longer decide from costs passed
to predict.
The optimal_threshold and optimal_rate of a MixtureMetric
no longer average its components' rates. A mixture of components of one strategy type that decide
from their costs alone, such as Cost, takes them from the weighted
combined costs. A mixture without Capability.OPTIMAL_RATE or Capability.OPTIMAL_THRESHOLD,
such as one of a Profit and a Savings component,
raises a ValueError.
MixtureMetric rejects components with different optimization
directions, and components that are amounts of money mixed with ones that are pure numbers, such
as AUEPC. Weights must not be negative and must sum to 1: constant
weights are checked when the mixture is built, the others when it is called.
A MixtureComponent weight can be an expression, such as '1 - success_rate - default_rate', whose parameters are parameters of the mixture. They can be
passed to a model's fit and are reported when missing. A callable weight is deprecated. empcs_score uses an expression weight, so it and the models trained on it
can be pickled.
[constrain>](https://empulse.readthedocs.io/en/stable/reference/generated/MixtureMetric.constrain <empulse.metrics.MixtureMetric.html#MixtureMetric.constrain <empulse.metrics.MixtureMetric.constrain>) rejects a lower
bound above the upper one, a name that is not a parameter, and a stochastic variable, which was
accepted and never checked. Called twice, it keeps the tighter bound. The bounds and conditions
also check a component's default when the parameter is not passed, unless the components have
different defaults.
A metric accepts a parameter under its symbol name or any of its aliases wherever it takes
one, and set_default and constrain can be called before alias. In a MixtureMetric, a bound on
an alias was bypassed by passing the symbol, a predicate raised a KeyError for a parameter
passed by its alias, and fixed parameters and defaults applied under one spelling only. A mixture
no longer warns about a parameter that only another component uses. A constraint on a name that is
neither a symbol nor an alias raises instead of being ignored.
[mark_outlier_sensitive>](https://empulse.readthedocs.io/en/stable/reference/generated/CostMatrix.mark_outlier_sensitive <empulse.metrics.CostMatrix.html#CostMatrix.mark_outlier_sensitive <empulse.metrics.CostMatrix.mark_outlier_sensitive>)
accepts an alias, finds a symbol created with assumptions such as positive=True, and works for
a symbol with several aliases, which RobustCSClassifier could not use. An
outlier-sensitive parameter is reported under its symbol name, and one that only enters a
distribution is classified by the costs that distribution enters. A parameter that components of a MixtureMetric mark for different classes is checked for outliers on every
sample, and one that every component fixes is no longer outlier-sensitive.
The optimal_threshold of Cost, Profit, Savings and LogCost is a single number when a
parameter is an array of length 1, which applies to every sample, instead of an array of one.
Scoring the empirical strategies is up to a quarter faster, and scoring Cost, Savings and LogCost gains another tenth on top of that. This speeds up the models
that score them on every iteration.
Copying a metric, as cloning or fitting a model with it as loss does, reuses its
compiled expressions instead of compiling them again. Cloning CSLogitClassifier(loss=empc_score) is 2.5 times as fast.
parameter_names lists every name a caller can pass, aliases
included. It lists a stochastic variable's distribution parameters instead of the variable, which
is drawn rather than passed. A model's set_fit_request no longer offers the variable either.
Building a Metric raises a ValueError when the cost matrix
names an alias after another of its symbols, which could then not be passed under its own name. It
also raises for a default or an outlier-sensitive marker on a stochastic variable, and for an
outlier-sensitive marker on a name that is neither a symbol nor an alias, which surfaced only as a
misleading error when fitting a RobustCSClassifier.
The parameters of a stochastic variable's distribution are checked for reserved and duplicate
names like any other symbol, instead of failing with an unclear TypeError or a SyntaxError
from sympy.
Metric and MixtureMetric raise a TypeError when passed sample_weight, for instance by a scorer that requests it through
metadata routing. They accepted it and ignored it, so a weighted cross-validation score was the
unweighted one. The costs of the cost matrix are what weigh each sample: fold any extra weight into
them, for example as an instance-dependent parameter. A cost-matrix parameter named sample_weight is still accepted.
Savings, savings_score and expected_savings_score rank models correctly when the baseline cost is
negative, as it is for a matrix of benefits: savings are now divided by the magnitude of the
baseline cost. A baseline that costs nothing returns nan with a warning instead of a huge number.
The 'prior' baseline of Savings predicts the positive class
prior, as documented, and a scalar or list baseline is accepted.
[optimal_threshold>](https://empulse.readthedocs.io/en/stable/reference/generated/LogCost.optimal_threshold <empulse.metrics.LogCost.html#LogCost.optimal_threshold <empulse.metrics.LogCost.optimal_threshold>) and optimal_rate are the Bayes-optimal decision for the scores of a model trained on the log cost,
0.5 whenever the cost matrix targets some but not all samples, instead of reusing Cost's threshold for calibrated probabilities.
When targeting nobody is optimal, optimal_threshold and classification_threshold return inf instead of 1.0, which still
targeted a probability of exactly 1.0 and some samples' decision scores. Cost, Profit, Savings, LogCost, CSThresholdClassifier, cost_loss and savings_score no longer invert the decision when the costs of the wrong
decisions are smaller than those of the right ones, as with fp_cost=1 and tp_cost=3. Costs
that make targeting pay off the less likely a sample is to be positive, which no threshold can
express, raise a ValueError.
Cost, Profit, Savings and LogCost no longer raise in optimal_threshold and optimal_rate when, for some sample, the costs do not depend on the
predicted probability. Such a sample is always targeted if that costs less and otherwise never,
including when both decisions cost the same.
A cost-matrix symbol named after a function its expression calls, such as a Weibull scale
named gamma, no longer breaks the metric.
An instance-dependent distribution parameter is validated when another parameter of the same
distribution is a scalar or a single-element list or array, e.g. a negative alpha array beside
a scalar beta.
Metrics accept boolean labels.
MaxProfit and MinCost average each
instance-dependent cost and benefit over the samples, rather than the parameters they are computed
from, so a cost such as the product of two instance-dependent parameters is averaged correctly. CSThresholdClassifier learns its single threshold from the same averaged
costs for any loss, MixtureMetric included, and ProfLogitClassifier with a gradient-free optimizer optimizes the same
score. A stochastic cost that is not linear in instance-dependent parameters warns that it is
approximated.
MaxProfit with stochastic variables finds the best threshold when
targeting loses money. When targeting either class loses, the optimum jumps between targeting
everybody and nobody. When targeting a positive loses and targeting a negative gains, it searched
only the part of the ROC convex hull above its diagonal. Other cost matrices are scored as before.
MaxProfit integrates numerically wherever a stochastic variable's
mass lies. With integration_method='quad', or a single variable whose expected profit has no
closed form, a distribution far from zero, such as a Laplace(200, 2), or much narrower or wider
than one gave about zero without a warning. Concentrated distributions, such as a Logistic or a
Gamma, Chi-squared or Beta with large shape parameters, no longer give nan or raise an OverflowError. This fixes empc_score with a concentrated Beta prior. A
large estimated integration error warns.
MaxProfit finds every point where the optimal threshold changes
for a profit that is not a polynomial in its stochastic variable, such as exp(v / 2), which
gave nan or a wrong optimal rate.
MaxProfit with a discrete stochastic variable, such as a Poisson or
a Bernoulli one, uses Monte Carlo integration under integration_method='auto', and 'quad' raises a clear error, instead of failing with an IndexError or an AttributeError. 'quasi-monte-carlo' with a distribution it has no quantile function for
raises a clear error instead of a KeyError.
MaxProfit with integration_method='monte-carlo' gives the same
result for the same distribution parameters, whichever were evaluated before. Under 'monte-carlo' and 'quasi-monte-carlo', the optimal rate is computed on the same samples as
the score, so the two describe the same operating point. Results for a given random_state
change.
A string term of a CostMatrix can no longer run Python code,
which made it unsafe to build a cost matrix from untrusted input. A string term may only use
names, numbers, arithmetic and sympy's mathematical functions such as exp or Max; build
anything else, such as sympy.Rational(1, 2), from sympy objects.
The LaTeX of Savings shows the savings relative to the baseline
instead of the cost divided by it.
Strategies that reduce a stochastic variable, such as Cost and EmpiricalMaxProfit, take the expectation of a term that is not linear in
it, such as gamma ** 2, instead of evaluating the term at the variable's mean. A term that is
not a polynomial in it, such as exp(gamma), warns that it is approximated.
[cost_matrix>](https://empulse.readthedocs.io/en/stable/reference/generated/Metric.cost_matrix <empulse.metrics.Metric.cost_matrix>.html#Metric.cost_matrix <empulse.metrics.Metric.cost_matrix>) returns a copy: a metric is
compiled from its cost matrix when it is built, so changing the matrix afterwards had no effect on
scoring. Build a new metric from the changed cost matrix instead.
MetricStrategy.requires_dynamic_boost_objective is removed. Use Capability.PRECOMPUTED_BOOST_OBJECTIVE not in strategy.capabilities instead.
Metrics, strategies and training objectives pickled with 0.13 or earlier are no longer
supported. Build or fit them again with this release before pickling them.
MaxProfit is faster with stochastic variables. A profit of
degree three or more is scored up to a third faster, the analytic gradients that ProfLogitClassifier and CSBoostClassifier train
with are 10 to 20% faster, and the optimal rate up to 15% faster. optimal_threshold of a metric
without stochastic variables is up to twice as fast.
The optimal rate of a MaxProfit metric with one stochastic
variable under integration_method='auto' is exact for every distribution that 'quasi-monte-carlo' supports, such as Laplace and Logistic, and two to three times as
fast for those.
The docstrings of max_profit_score and MaxProfit correct the formula and say how instance-dependent parameters
are averaged.
Models
ProfLogitClassifier, ProfTreeClassifier, ProfSRClassifier, ProfMPMClassifier and ProfMEMPMClassifier take a threshold parameter. With the default 'auto', predict predicts the positive class above threshold_, the probability that costs
the training samples least, instead of above 0.5. Across nine benchmark datasets this raised the
share of attainable savings on held-out data by 0.03 to 0.33 on average, depending on the model.
Pass threshold=0.5 for the previous predictions. See User Guide.
CSThresholdClassifier and CSRateClassifier
take a cv parameter to learn a decision that depends on the scores, such as a MaxProfit threshold or any rate, from out-of-fold scores instead of the
scores of the training rows, which flatter an estimator that overfits them. A threshold that
depends only on the costs ignores it.
CSForestClassifier accepts min_impurity_decrease=None, as CSTreeClassifier does, and both it and the default tree of CSBaggingClassifier now use it: a tree only splits while that lowers the
cost of its training samples, instead of growing until its leaves are pure. On the 15 bundled
datasets the ensembles' own predictions save more, with a tenth of the leaves or fewer and fits
taking half the time. Pass min_impurity_decrease=0.0 for the previous trees.
CSLogitClassifier and ProfLogitClassifier
raise a ValueError before fitting when their optimizer needs a gradient the loss has no
objective for, such as LBFGSBOptimizer with an EmpiricalMaxProfit metric, and the message names an optimizer that works.
They raised a NotImplementedError from inside the fit.
The baseline of a Savings loss can be passed to the fit of
cost-sensitive models and routed through metadata routing like the cost matrix's parameters, also
in a nested MixtureMetric whose components use different strategies. Most
models raised a TypeError, a nested mixture ignored it with an unknown-parameter warning, and CSThresholdClassifier and CSRateClassifier passed it
on to their estimator, so a mixture of savings always trained against the 'zero_one' baseline.
A loss parameter passed to fit as a list or a pandas Series is treated like an array. CSThresholdClassifier learned one threshold per training sample from it,
so predict failed on other data or applied the wrong thresholds, and weighted voting in CSForestClassifier and CSBaggingClassifier raised
a length mismatch.
Cost-sensitive models accept an array fit parameter whose length is not the number of
samples, such as a per-feature weight, when it is not a cost or loss parameter.
After set_params(loss=...), set_fit_request accepts the new loss's parameters.
CSThresholdClassifier and CSRateClassifier
without a loss keep each cost not passed to predict at its value in fit, instead of the
constructor's, which predicted every sample positive after fit(X, y, fp_cost=5, fn_cost=10)
and predict(X, fn_cost=10). A predict-time cost with a length other than the samples' is
rejected with a clear error. Fitted without costs, or with every cost zero even over non-zero
constructor costs, they learn no decision, forget the decision of an earlier fit, and fit the same
calibrated estimator as with costs.
CSThresholdClassifier's default calibrator passes the costs on to a
cost-sensitive estimator, which was trained on fp_cost=fn_cost=1 instead, and with metadata
routing it is fitted with the sample weights the estimator requests, which raised before.
CSThresholdClassifier and CSRateClassifier
with pos_label set to the smaller class pass a cost-sensitive estimator the costs as it sees
them, with the true and false positives and negatives swapped, instead of inverting its training.
CSRateClassifier predicts exactly ceil(rate_ * n_samples) samples
positive. Tied scores at the cut-off made it predict more, up to five times the rate on a shallow
tree's scores, and rounding error one more for rates such as 0.07 of 100 samples. Its documentation
now says that the rate applies to each predict batch.
RobustCSClassifier standardizes the residuals of the costs, so outlier_threshold is a number of standard deviations. It divided the absolute residuals by
their own standard deviation, as the original paper does, so the default of 2.5 flagged around 13%
of normally distributed costs instead of 1%.
RobustCSClassifier measures and imputes the outliers of a cost that
only enters one class's rows, such as fn_cost, among that class's rows. The other class's
values widened the spread, so real outliers were missed and rows the cost never uses were imputed.
It also detects outliers on the positive class of a wrapped estimator's pos_label, and accepts
an outlier-sensitive parameter given as a list.
RobustCSClassifier wraps a Pipeline whose
last step is a cost-sensitive classifier. It looked for the loss and the costs on the pipeline itself,
so it missed a loss set on the classifier, and passed the costs to Pipeline.fit under their
plain names, which it rejects. Without metadata routing, the costs now go to the last step as <step>__<cost>; with it, to the steps that request them.
RobustCSClassifier keeps the costs set on the wrapped estimator
when neither fit nor its own constructor sets them, instead of overriding them with zero.
CSTreeClassifier and CSForestClassifier
accept a class_weight dict keyed by the class labels, as documented. The dict was looked up by
the 0/1 encoding of the labels, so only the labels 0 and 1 worked. A key that is no class raises a
clear error, so a dict keyed by 0 and 1 for other labels has to be keyed by the labels instead.
The criterion="gini" impurity of CSTreeClassifier and CSForestClassifier is per unit of weight, like "cost" and "entropy". It grew with node size, which favoured even splits and let noise pass min_impurity_decrease. Trees grown with it change.
A node of CSTreeClassifier, CSForestClassifier
and CSBaggingClassifier that holds a single class is a leaf, unless its
samples disagree on which prediction costs less. With negative or non-zero correct-decision costs
such nodes kept splitting without changing any prediction.
Weighted voting in CSForestClassifier and CSBaggingClassifier weighs each estimator by how much better than average
its out-of-bag loss is, relative to the average cost of a wrong decision, and gives a tree without
out-of-bag samples the average weight. The worst estimator always got no vote, so of two nearly
equal trees only one counted, and a tree without out-of-bag samples made the forest weigh every tree
alike and bagging fail.
CSForestClassifier passes a callable oob_score the class labels,
as scikit-learn's forests do, instead of their 0/1 encoding.
CSBaggingClassifier fits when a bootstrap sample holds only one
class, which then gives a tree of a single leaf. It raised a ValueError.
CSBaggingClassifier with warm_start=True keeps the estimators
fitted earlier and only fits the new ones. It refitted all of them.
CSBaggingClassifier raises a clear error for a base estimator whose fit does not accept the costs.
CSBoostClassifier and B2BoostClassifier
encode the labels of an eval_set passed in fit_params as they encode y. String labels
crashed XGBoost, and labels such as 1/2 or -1/1 were scored as if they were 0/1.
ProfTreeClassifier raises a clear error when every variation operator
rate is zero, instead of a ZeroDivisionError. The rates are documented as the relative weights
they are: they need not sum to 1.
ProfTreeClassifier reports n_iter_ correctly when stopping early,
and reads min_samples_split=1.0 and min_samples_leaf=1.0 as fractions of the samples.
BiasReweighingClassifier weighs every sample alike, with a warning,
when the sensitive feature has a single value, as the resampling and relabeling models do. With
only the value 0, the weights followed the class prior instead.
Bias mitigation and samplers
BiasRelabelingClassifier passes its fit parameters, such as the
parameters of a Metric loss, to the estimator that ranks the relabelling
candidates too.
The bias mitigation models and samplers raise a clear error for a sensitive_feature that
is not coded as 0 and 1 (after transform_feature), or whose length differs from y, instead
of failing with an index error or silently doing nothing.
BiasRelabler without a sensitive_feature returns the data
unchanged, like BiasResampler, and relabels a pandas target with any index.
BiasResampler and BiasResamplingClassifier
keep the number of samples and no longer return the resample ordered by class and group. Results
for a given random_state change.
The fetch_* loaders only download over https. A http, ftp or file URL, such
as one returned by the UCI or OpenML API or reached through a redirect, raises an error instead.
Packaging and dependencies
Support for Python 3.15, including the free-threaded build (cp315t). It needs
numpy 2.5.2, scipy 1.18.1, scikit-learn 1.9.1 and, for the datasets extra, pyarrow 26.0 or
later. CatBoost has no Python 3.15 wheels yet, so there the boosting extra installs XGBoost
and LightGBM only. The polars backend of datasets needs polars 2.0 on Python 3.15.
Building Empulse from source requires Cython 3.1.2 or later. Cython 3.1.0 and 3.1.1 built
a CSBoostClassifier and B2BoostClassifier that
failed to fit or pickle with XGBoost.