Add smoothing to mean encoder

[glevv]

Will solve #315

• add tests
• add info to docs

[solegalli]

Hi @glevv

Thank you for the PR. I missed that you marked it ready for review for a few days because it still said WIP, and I didnt look all the way through the history :/ . And then I was a bit busy with a deadline that I had for yesterday.

I will look at this tomorrow.
Apologies for the delay.

[solegalli]

Hi @glevv

Thank you so much for taking care of this enhancement. The code looks great overall.

I think it would be important to change the parameter name and add the formula in the docstrings and link the formula with the parameter.

Also, I think there is a bit, if I remember correctly, that is missing?

If a category is rare, they would be replaced by the overall mean of the target. Otherwise, they will be replaced by a blended posterior and prior probability that gives more weight to the posterior the more observations there are in the category.

There is a threshold to consider a category rare, that in category encoders is handed by an additional parameter.

And then the blend in the probabilities are weighted by the number of observations in the category. If a category has a lot of observations, more weight is given to the posterior, alternatively to the prior.

In the current implementation, we are not weighting the probability blend, right? unless a is auto.

Let me know f I am not understanding something, and if we could add the weighting?

Thank you!

[solegalli]

In category encoders, the smoothing parameter is weighted by the number of observations per category:
https://github.com/scikit-learn-contrib/category_encoders/blob/80288a0aa3d9bc4406f3d6df13a49113bbc3fedf/category_encoders/target_encoder.py#L229-L231

[glevv]

I did not use sigmoid formula, I used (x / (x+smoothing)) formula (formula 8 in Micci-Barreca paper), which does not have this parameter, only smoothing factor.

The only way we could implement min_samples is by manually filtering categories with less number of observations and assigning them y_prior, which I'm not a fan of, since it will raise some questions in the future

[solegalli]

Oh, I see, so damping = y.groupby(X[var]).var(ddof=0) / y_var is equation 6 in the article and x / (x+smoothing) is equation 5, am I right? Both in section 3.1

Any particular reason you chose this implementation?

I think it might be useful to offer some guidance regarding what are good values of a, if they exist. Or is it just by cross-validation and hyperparamter search?

[glevv]

I choose this formula for three reasons:

1. Almost all implementations of target encoding out there use sigmoid formula (category_encoders, dirty_cat etc), and it's meaningless to just reimplement it for 100th time.
2. This formula has an interesting auto option that I've never encountered in target encoding estimators (or maybe I searched not good enough, idk), so it seemed interesting and unique.
3. This formula allow us to set smoothing to 0 and get vanilla mean encoder, which is great for backwards compatibility and dependencies. Sigmoid formula can't do it.

As for meaningful smoothing values, I think in general it's goes like this "the more noise you have in target, the bigger value of smoothing should be" same goes for high cardinality, but there is no advice on concrete values. Most of the time its optimized by cv, sometimes its just set as 1. Trick here is that we will have an auto option for that.

[solegalli]

Makes sense. Thank you!

[solegalli]

FYI @glevv

glevv#1

Other than that, this is good to go. Thank you so much!