ENH: concat and append now can handle unordered categories

[sinhrks]

• closes Appending Pandas dataframes in for loop results in ValueError #13524
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

on current master:

# different categoricals
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')])
# ValueError: incompatible categories in categorical concat

# categorical + normal (values are contained in categories) -> object dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([1, 2])])
#0    1
#1    2
#0    1
#1    2
# dtype: object

# categorical + normal (values are not contained in categories) -> object dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4])])
#0    1
#1    2
#0    3
#1    4
# dtype: object

this PR (updated according to the discussion):

# different categoricals
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')])
#0    1
#1    2
#0    3
#1    4
# dtype: int64

# specifying union_categoricals keeps category if possible
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')], union_categoricals=True)
#0    1
#1    2
#0    3
#1    4
# dtype: category
Categories (4, int64): [1, 2, 3, 4]

# categorical + normal (values are contained in categories) -> category dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([1, 2])])
#0    1
#1    2
#0    1
#1    2
# dtype: int64

# categorical + normal (values are not contained in categories) -> int dtype (keep original dtype)
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4])])
#0    1
#1    2
#0    3
#1    4
# dtype: int64

CC: @JanSchulz, @chris-b1

[codecov-io]

Current coverage is 85.25% (diff: 100%)

Merging #13767 into master will decrease coverage by <.01%

@@             master     #13767   diff @@
==========================================
  Files           139        139          
  Lines         50496      50491     -5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43049      43044     -5   
  Misses         7447       7447          
  Partials          0          0          

Powered by Codecov. Last update 8023029...96a372e

[jankatins]

IMO the old test case should be modified that it still again tests the case that "more unequal" cats fail the concat.

[jankatins]

What is the actual "problem" or use case here?

I'm opposed to this PR: IMO concat() should not change the categories. In my thinking (pd.Series([1, 2], dtype='category'), pd.Series([3, 4], dtype='category') is as (un)similar as ([1,2], ["A", "B"]). The latter might cast to a type which can take both (object) but not to a new int_or_string type...

So, I find the master variant more in line with that thinking. What I'm not sure is whether it should raise or (as it does now) convert to object. Other cat handling of two different cats (apart from the special union_categoricals) raises but pandas in general casts to object. As cat handling deviates in other cases from normal pandas casting rules, IMO it should error here, too.

I can see a logic for "categorical + iterable of the same values as in the categories = new categorical with the same categories as the first one" as that can be thought similar as cat[x] = value.

[sinhrks]

@JanSchulz thx for sharing your opinion. Can you illustrate your intention covering all possible patterns? (I copied mine from #13524).

• concat 2 categories -> use the rule of union_categorical
• concat category and other dtype (which values are all in the category, including empty) -> category
  • this rule is applied regardless of order (if there is at least one category in concatenating values)
  • the property like ordered should be preserved.
• concat category and other dtype (which values are not in the category) -> not category (dtype is infered)

[jankatins]

I come from using catagoricals as things like lickert scales: "Fully agree ... full disagree" and such things. From that standpoint, I usually read data in, convert to cat and replace the categories with the full set. In that case the categories have a specific meaning (e.g the lickert scale or in other cases the 50 US states or such things which are fixed).

From that comes the rule that you shouldn't be able to set a value in the cat column to something not in categories (adding "Germany" to a cat encoding the 50 US states makes no sense). And from that the "error if combining two different categories" rule. So I'm fully happy that there isn't any "default" (i.e. anything in the cat accessor or in the default "non-categorical-specific" API like concat and similar functions) way to combine two different categories and trying to do it would error.

I see union_categorical as a way to use a categorical as a memory saving string type and as in the other issues where such additions/changes were proposed, I would love to see such a type instead of changing categorical to fit that needs.

The second usecase is as an internal helper for someone using it with reading in csv files before letting the user do some "meaningful" replacements.

[sinhrks]

Thanks for sharing the usecase. Based on this, what the "rules" on your mind are? can u list up possible rules like: category + category (same category) = category covering possible patterns to discuss the expected behavior.

[jankatins]

IMO these are the rules:

• category + setting with values in the categories = fine
• category + setting with values not in the categories = error (not converting to object/common "normal" type)
• category + same category configuration ("order" + categories OR "unordered" + same categories (the unordered case could actually be less strict and only test if the set of categories is the same)) = fine
• category + different category = error

[jreback]

I agree with @JanSchulz rules, except where error -> just coerce to the base type (of the categorical) if there is one, if multiple categoricals and different base types then must upcast (potentially to object).

The idea is that concat/append would always work, whether you add new categoricals or not, but the dtype will be preserved in a strict way.

I suppose that we could further add an argument to concat itself that just calls union_categoricals, IOW allows a non-strict interpretation. maybe union_categoricals=False as the default

[jankatins]

@jreback

category + setting with values not in the categories = error (not converting to object/common "normal" type)

except where error -> just coerce to the base type (of the categorical) if there is one, if multiple categoricals and different base types then must upcast (potentially to object).

If you also want to change setting as well, then this is a whole different thing... IMO the above rules are what currently applies.

I'm still not convinced: So what happens if I have a Categorical which is encoded as complete agree ... completely disagree (5 step lickert scale) and now I concat a column which is encoded as a categorical which encode the 50 US States. It will now work (both are string dtypes), but the meaning is completely screwed. Or imagine things like two categoricals which have one category in common (50 US states + 50 states capitals -> washington is in both...): will that mean that the final categorical have x+y-1 categories? In my opinion, each categorical configuration is it's own dtype (is_dtype_equal in Categorical actually tests just that: https://github.com/pydata/pandas/blob/master/pandas/core/categorical.py#L1791-L1809), so upcasting to a combined one is the same as removing all meaning from the categorical configuration = upcasting to object.

What is the actual usecase here: internal csv parsing of different files to a single dataframe? If so, then in my eyes this looks (again :-)) like the "pd-String" usecase: you encode it directly to save memory and then you are have the problem to concat two dfs because you don't know the strings (=you have no beforehand knowledge about the expected categories).

If so, then there are multiple different ways to solve this:

• add a pd.String dtype :-)
• add a convenience method like pd.concat_nonstrict() which would basically do a union_categorical on the category columns.
• add a nonstrict_category_checks=False to concat/append which would basically switch to pd.concat_nonstrict() behaviour above.

[sinhrks]

I agree that concat shouldn't fail, as users understand what they're doing. Considering all dtypes (not only category), coercing to the object dtype should be a basic rule (such as DatetimeTZ with different tz).

Based on this, I've organized the rules as below table.

left	right	result
category	category (identical categories and ordered)	category
category	category (different categories or ordered)	object (dtype is inferred)
category	not category	object (dtype is inferred)

Also, adding following options to concat should cover all cases:

• union_categorical (default False): work as @jreback suggested. Apply union_categorical rule to concat Categorical.
• strict (default False): raise when different dtype / different categories are being concatenated.

[jreback]

categoricals

[sinhrks]

once updated based on the discussion. Note that following fixed are not included:

• union_categoricals with Index.append (being included in BUG: concat/append misc fixes #13660)
• strict option (should be after BUG: concat/append misc fixes #13660)

[jreback]

there is a doc-string in merge.rst, can you make sure to update that as well.

[jreback]

ok, let's take out union_categoricals arg to concat for now and put it in a separate PR, @sinhrks ?

[wesm]

@jreback that sounds good to me. Easier to add it later, and having the extra function is a stopgap for users who need it

[jreback]

can you add a link with a note in merge.rst to here

[jreback]

@sinhrks can you update

[jreback]

@sinhrks can you update?

[jorisvandenbossche]

@sinhrks I think the decision was to take out the kwarg for now? (see #13767 (comment) above).
(but maybe that is quite some work ..)

[sinhrks]

@jorisvandenbossche You're right. Updated to remove union_categoricals kw.

[jreback]

can u add simikar for Datetimes and all NaT ?

[sinhrks]

sure, will do in today.

[jorisvandenbossche]

@sinhrks @jreback I am going to merge this, the final comments can be addressed in a follow-up PR (but they are not critical for the rc I think).