Rethinking the CategoricalEncoder API ?

[jorisvandenbossche]

Based on some discussions we are having here and issues that are opened, we are having some doubts that CategoricalEncoder (#9151) was the good choice of name (and since it is not released yet we have some room for change).

So summary of how it is now:

• The class name CategoricalEncoder says what type of data it accepts (categorical data)
• The keyword argument encoding specifies how to encode those data

Currently we already have encoding='onehot'|'onehot-dense'|'ordinal'.

But what to do in the following cases:

• We want to add more encoding options (eg binary encoding, mean target encoding, unary encoding, ...). Do we keep adding those as new values for the encoding kwarg in the one big CategoricalEncoder class?
• We want to add an option specific to one of the encodings (eg for 'onehot' encoding to drop the first (redundant) column, or for 'ordinal' encoding base the order of the categories on the frequency, ...). The problem here is that we then need to add additional keyword arguments to CategoricalEncoder that are or are not active depending on what you passed for encoding kwarg, which is not the nicest API design.

For that last problem, we already had this with the sparse=True/False option, which was only relevant for 'onehot' and not for 'ordinal', and which we solved with having both 'onehot' and 'onehot-dense' encoding options and not a sparse keyword. But such an approach also does not scale.

Related to this, there is a PR to add a UnaryEncoder (#8652). There was a related discussion on the naming in that PR, as currently the name says how it encodes, not what type of data it gets (in the current design, it accepts already encoded integers, not actual categorical data. In that regard, to be consistent with CategoricalEncoder, it might better be named OrdinalEncoder because it needs ordinal data as input).

---

What are the options forward:

1. Keep things as we have it now in master, and be be OK with adding some new options to the single class (an important question which is hard to answer now, is how much new features we will want to add in the future).
2. Switch the naming scheme and have a bunch of 'categorical encoders' where the name says how it encodes (OnehotEncoder, OrdinalEncoder, and later maybe BinaryEncoder, UnaryEncoder, ...)

So it is a bit a trade-off of potential build up of number of classes vs number of keyword arguments in a single class.

---

One problem with the second approach (and one of the reasons we went with CategoricalEncoder in the first place, even before we added the multiple encoding options), is that there is already a OnehotEncoder, which has a different API than the CategoricalEncoder. And, there is not really a good other name we could use for the encoder that does one-hot encoding.
However, I think that, with some temporary ugly hacks, we could reuse the name, if we are OK with deprecating the current attributes (and I think we agree it are not the most useful attributes). The idea would be that if you fit the class with string data, you get the new behaviour, and if you fit the class with integer data, you get a deprecation warning indicating the default behaviour will change (and indicating which keyword to specify to get rid of the warning).

cc @jnothman @amueller @GaelVaroquaux @rth

[ogrisel]

Thanks for the summary @jorisvandenbossche. I think I am in favor of option 2: reuse OneHotEncoder class, deprecate the weird attributes and add a constructor parameter to select the behavior with a future warning that says that the default behavior will change but makes it easy to silence that warning just by passing a value for that option.

[jnothman]

The idea of reverting CategoricalEncoder makes me quite sad, but I think you're right that future users would be less mystified by option 2. My main concern is that we have tried implementing this as a change to OHE for a long time and it never flew. Perhaps it would be good to attempt the modifications to the OneHotEncoder docstring according to that proposed change, so we can see if it looks sane.

[GaelVaroquaux]

+1 to what Joel said ⁣Sent from my phone. Please forgive typos and briefness.​

…

On Jan 23, 2018, 12:28, at 12:28, Joel Nothman ***@***.***> wrote: The idea of reverting CategoricalEncoder makes me quite sad, but I think you're right that future users would be less mystified by option 2. My main concern is that we have tried implementing this as a change to OHE for a long time and it never flew. Perhaps it would be good to attempt the modifications to the OneHotEncoder docstring according to that proposed change, so we can see if it looks sane. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #10521 (comment)

[jorisvandenbossche]

The idea of reverting CategoricalEncoder makes me quite sad

To be clear, it would not be a revert, it would be a refactor / rename that keeps all functionality!
But I also like the "CategoricalEncoder" name, that would indeed be sad.

That said, I will quickly try to do the changes to have an idea how possible it is to integrate this in OnehotEncoder.

[jorisvandenbossche]

OK, I opened a PR with a proof of concept: #10523.
It's not yet complete (no deprecation warnings and new attributes are not yet calculated in old behaviour).

The main API question is about the format of the input data.
So as a recap, there are two different ways we currently process the categorical data:

1. As actual, not yet encoded (integer or string), categorical data (how it is done in CategoricalEncoder) -> infer categories from the unique values in the training data
2. As integer, already encoded data (how it is done in the current OneHotEncoder) -> infer categories from the maximum value in the training data

The question is: do we find both cases worth supporting? Thus, in the potentially merged OneHotEncoder, do we keep the ability to do both, or do we fully deprecate and then remove the ability to process ordinal input?

If want the ability to process both, we can add a boolean keyword to specify the input data type (for now I use encoded_input=False/True, but other ideas are ordinal_input, ...)

For the deprecation period, we have to support both anyway, and also have to introduce a keyword to choose the behaviour (to be able to silence the warning and choose the new behaviour).
So in principle we could just keep the keyword afterwards.

Given that we want to handle both, an overview of how OneHotEncoder would work:

• for now encoded_input=None, and we infer the default based on the data
• if int-like data (handled before by OneHotEncoder) encoded_input is internally set to True and a deprecation warning is raised. If the user wants to keep the current behaviour, it can manually specify it as OneHotEncoder(encoded_input=True) to silence the warning.
• if the input is not int-like, we set encoded_input internally to False and without warning use the new behaviour (= the current CategoricalEncoder behaviour)
• in the future we change the default of encoded_input from None to False (by default the new behaviour, also for int-like data)

[jnothman]

I'm still not sure what you're suggesting is the practical difference due to inferring categories from the max value.

[jorisvandenbossche]

@jnothman I suppose you acknowledge there can be a difference in practice? (the output you get depending on the data you have)

But whether this difference is important in practice, I don't know. That's where I would like to see feedback. Whether anybody actually wants this "max value"-based method, or whether we are fine with (in the future, after deprecation) only having the "unique values"-based method.

I think I personally would never need this max-value based method, but the OneHotEncoder has been like that for many years (for good reason or not?).

Actually deprecating the max-value based categorization would certainly make the implementation (after deprecation) simpler.
And if we choose for that route, I agree the option should rather be legacy_mode=True/False rather than encoded_input/ordinal_input

[jnothman]

Remind me what the actual difference in output is, when n_values='auto', please? I had thought the active_features_ thing made them basically identical, but I'm probably forgetting something.

[jorisvandenbossche]

Aha, that clarifies our misunderstanding :-)
I misunderstood how the current OneHotEncoder is actually working. Suppose you have one feature with values [2, 3, 5, 2]. I thought the current OneHotEncoder would have categories [0, 1, 2, 3, 4, 5] (while current CategoricalEncoder would have categories [2, 3, 5]). But you are right that the active_features_ is also only [2, 3, 5], essentially making them the same with the default value of n_values='auto'.

So it is only the case where you pass an integer to n_values (like n_values=6 for categories=[0, 1, 2, 3, 4, 5] in the above case) to specify the number of categories that will actually be an API change (deprecated / removed).
And that will be easily be replacable by the user with categories=range(6)

Sorry for the confusion.
In that light, I think we even don't need the legacy_mode option. We can just translate n_values=6 to categories=range(6) internally and raise a warning for that (but need to check this with the actual tests).

The other difference is the handling of unseen categories. With the current behaviour of the OneHotEncoder, if the unseen values are within the range(0, max), it will not raise an error even if handle_unknow='error' (the default). But also that can be solved separately by in such a case raising a warning that the user should set handle_unknown='ignore' manually to keep the existing behaviour.

The only feature we would loose is the distinction between unknown categories that are within the range(0, max) (by the current OneHotEncoder not regarded as 'unknown') and those that are bigger than that (> max, those are currently already regarded as unknown by the OneHotEncoder).

[jnothman]

no, that is the sort of thing we have tried before and it's just too finicky. unless there is good reason to maintain current behaviour, we should just have a legacy_mode to slowly bring us to the future.

[jorisvandenbossche]

no, that is the sort of thing we have tried before and it's just too finicky.

Can you clarify to which aspect this "no" refers?
To the fact that I think a legacy_mode is not needed?

[jnothman]

yes, to the idea that you can just make something that is both backwards compatible and what we want going forward

[jorisvandenbossche]

yes, to the idea that you can just make something that is both backwards compatible and what we want going forward

That was not what I tried to suggest. I wanted to make clear that think it is possible to not have a legacy_mode keyword, not by having it magically both backwards compat and what we want in the future, but by deprecating the behaviour of the existing keywords.

So to be concrete: a non-default value of n_values can be deprecated and has to be replaced by categories specification. handle_unknow in case of integer data should be set explicitly by the user to choose either full ignoring or full erroring instead of current mix (and otherwise deprecation warning is raised).