pandas
python
import numpy as np import pandas as pd np.random.seed(123456) np.set_printoptions(precision=4, suppress=True) pd.options.display.max_rows = 15
This is an introduction to pandas categorical data type, including a short comparison with R's factor
.
Categoricals are a pandas data type, which correspond to categorical variables in statistics: a variable, which can take on only a limited, and usually fixed, number of possible values (categories; levels in R). Examples are gender, social class, blood types, country affiliations, observation time or ratings via Likert scales.
In contrast to statistical categorical variables, categorical data might have an order (e.g. 'strongly agree' vs 'agree' or 'first observation' vs. 'second observation'), but numerical operations (additions, divisions, ...) are not possible.
All values of categorical data are either in categories or np.nan. Order is defined by the order of categories, not lexical order of the values. Internally, the data structure consists of a categories array and an integer array of codes which point to the real value in the categories array.
The categorical data type is useful in the following cases:
- A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory, see
here <categorical.memory>
. - The lexical order of a variable is not the same as the logical order ("one", "two", "three"). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order, see
here <categorical.sort>
. - As a signal to other python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).
See also the API docs on categoricals<api.categorical>
.
Categorical Series or columns in a DataFrame can be created in several ways:
By specifying dtype="category"
when constructing a `Series`:
python
s = pd.Series(["a","b","c","a"], dtype="category") s
By converting an existing Series or column to a category
dtype:
python
df = pd.DataFrame({"A":["a","b","c","a"]}) df["B"] = df["A"].astype('category') df
By using some special functions:
python
df = pd.DataFrame({'value': np.random.randint(0, 100, 20)}) labels = [ "{0} - {1}".format(i, i + 9) for i in range(0, 100, 10) ]
df['group'] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels) df.head(10)
See documentation <reshaping.tile.cut>
for ~pandas.cut
.
By passing a pandas.Categorical
object to a Series or assigning it to a DataFrame.
python
- raw_cat = pd.Categorical(["a","b","c","a"], categories=["b","c","d"],
ordered=False)
s = pd.Series(raw_cat) s df = pd.DataFrame({"A":["a","b","c","a"]}) df["B"] = raw_cat df
Anywhere above we passed a keyword dtype='category'
, we used the default behavior of
- categories are inferred from the data
- categories are unordered.
To control those behaviors, instead of passing 'category'
, use an instance of ~pandas.api.types.CategoricalDtype
.
python
from pandas.api.types import CategoricalDtype
s = pd.Series(["a", "b", "c", "a"]) cat_type = CategoricalDtype(categories=["b", "c", "d"], ordered=True) s_cat = s.astype(cat_type) s_cat
Categorical data has a specific category
dtype <basics.dtypes>
:
python
df.dtypes
Note
In contrast to R's factor function, categorical data is not converting input values to strings and categories will end up the same data type as the original values.
Note
In contrast to R's factor function, there is currently no way to assign/change labels at creation time. Use categories to change the categories after creation time.
To get back to the original Series or numpy array, use Series.astype(original_dtype)
or np.asarray(categorical)
:
python
s = pd.Series(["a","b","c","a"]) s s2 = s.astype('category') s2 s2.astype(str) np.asarray(s2)
If you have already codes and categories, you can use the ~pandas.Categorical.from_codes
constructor to save the factorize step during normal constructor mode:
python
splitter = np.random.choice([0,1], 5, p=[0.5,0.5]) s = pd.Series(pd.Categorical.from_codes(splitter, categories=["train", "test"]))
0.21.0
A categorical's type is fully described by
categories
: a sequence of unique values and no missing valuesordered
: a boolean
This information can be stored in a ~pandas.api.types.CategoricalDtype
. The categories
argument is optional, which implies that the actual categories should be inferred from whatever is present in the data when the pandas.Categorical
is created. The categories are assumed to be unordered by default.
python
from pandas.api.types import CategoricalDtype
CategoricalDtype(['a', 'b', 'c']) CategoricalDtype(['a', 'b', 'c'], ordered=True) CategoricalDtype()
A ~pandas.api.types.CategoricalDtype
can be used in any place pandas expects a dtype. For example pandas.read_csv
, pandas.DataFrame.astype
, or in the Series constructor.
Note
As a convenience, you can use the string 'category'
in place of a ~pandas.api.types.CategoricalDtype
when you want the default behavior of the categories being unordered, and equal to the set values present in the array. In other words, dtype='category'
is equivalent to dtype=CategoricalDtype()
.
Two instances of ~pandas.api.types.CategoricalDtype
compare equal whenever they have the same categories and orderedness. When comparing two unordered categoricals, the order of the categories
is not considered
python
c1 = CategoricalDtype(['a', 'b', 'c'], ordered=False)
# Equal, since order is not considered when ordered=False c1 == CategoricalDtype(['b', 'c', 'a'], ordered=False)
# Unequal, since the second CategoricalDtype is ordered c1 == CategoricalDtype(['a', 'b', 'c'], ordered=True)
All instances of CategoricalDtype
compare equal to the string 'category'
python
c1 == 'category'
Warning
Since dtype='category'
is essentially CategoricalDtype(None, False)
, and since all instances CategoricalDtype
compare equal to 'category'
, all instances of CategoricalDtype
compare equal to a CategoricalDtype(None, False)
, regardless of categories
or ordered
.
Using .describe()
on categorical data will produce similar output to a Series or DataFrame of type string
.
python
cat = pd.Categorical(["a", "c", "c", np.nan], categories=["b", "a", "c"]) df = pd.DataFrame({"cat":cat, "s":["a", "c", "c", np.nan]}) df.describe() df["cat"].describe()
Categorical data has a categories and a ordered property, which list their possible values and whether the ordering matters or not. These properties are exposed as s.cat.categories
and s.cat.ordered
. If you don't manually specify categories and ordering, they are inferred from the passed in values.
python
s = pd.Series(["a","b","c","a"], dtype="category") s.cat.categories s.cat.ordered
It's also possible to pass in the categories in a specific order:
python
s = pd.Series(pd.Categorical(["a","b","c","a"], categories=["c","b","a"])) s.cat.categories s.cat.ordered
Note
New categorical data are NOT automatically ordered. You must explicitly pass ordered=True
to indicate an ordered Categorical
.
Note
The result of Series.unique()
is not always the same as Series.cat.categories
, because Series.unique()
has a couple of guarantees, namely that it returns categories in the order of appearance, and it only includes values that are actually present.
python
s = pd.Series(list('babc')).astype(CategoricalDtype(list('abcd'))) s
# categories s.cat.categories
# uniques s.unique()
Renaming categories is done by assigning new values to the Series.cat.categories
property or by using the Categorical.rename_categories
method:
python
s = pd.Series(["a","b","c","a"], dtype="category") s s.cat.categories = ["Group %s" % g for g in s.cat.categories] s s.cat.rename_categories([1,2,3]) s # You can also pass a dict-like object to map the renaming s.cat.rename_categories({1: 'x', 2: 'y', 3: 'z'}) s
Note
In contrast to R's factor, categorical data can have categories of other types than string.
Note
Be aware that assigning new categories is an inplace operations, while most other operation under Series.cat
per default return a new Series of dtype category.
Categories must be unique or a ValueError is raised:
python
- try:
s.cat.categories = [1,1,1]
- except ValueError as e:
print("ValueError: " + str(e))
Categories must also not be NaN
or a ValueError is raised:
python
- try:
s.cat.categories = [1,2,np.nan]
- except ValueError as e:
print("ValueError: " + str(e))
Appending categories can be done by using the Categorical.add_categories
method:
python
s = s.cat.add_categories([4]) s.cat.categories s
Removing categories can be done by using the Categorical.remove_categories
method. Values which are removed are replaced by np.nan
.:
python
s = s.cat.remove_categories([4]) s
Removing unused categories can also be done:
python
s = pd.Series(pd.Categorical(["a","b","a"], categories=["a","b","c","d"])) s s.cat.remove_unused_categories()
If you want to do remove and add new categories in one step (which has some speed advantage), or simply set the categories to a predefined scale, use Categorical.set_categories
.
python
s = pd.Series(["one","two","four", "-"], dtype="category") s s = s.cat.set_categories(["one","two","three","four"]) s
Note
Be aware that Categorical.set_categories
cannot know whether some category is omitted intentionally or because it is misspelled or (under Python3) due to a type difference (e.g., numpys S1 dtype and python strings). This can result in surprising behaviour!
If categorical data is ordered (s.cat.ordered == True
), then the order of the categories has a meaning and certain operations are possible. If the categorical is unordered, .min()/.max()
will raise a TypeError.
python
s = pd.Series(pd.Categorical(["a","b","c","a"], ordered=False)) s.sort_values(inplace=True) s = pd.Series(["a","b","c","a"]).astype( CategoricalDtype(ordered=True) ) s.sort_values(inplace=True) s s.min(), s.max()
You can set categorical data to be ordered by using as_ordered()
or unordered by using as_unordered()
. These will by default return a new object.
python
s.cat.as_ordered() s.cat.as_unordered()
Sorting will use the order defined by categories, not any lexical order present on the data type. This is even true for strings and numeric data:
python
s = pd.Series([1,2,3,1], dtype="category") s = s.cat.set_categories([2,3,1], ordered=True) s s.sort_values(inplace=True) s s.min(), s.max()
Reordering the categories is possible via the Categorical.reorder_categories
and the Categorical.set_categories
methods. For Categorical.reorder_categories
, all old categories must be included in the new categories and no new categories are allowed. This will necessarily make the sort order the same as the categories order.
python
s = pd.Series([1,2,3,1], dtype="category") s = s.cat.reorder_categories([2,3,1], ordered=True) s s.sort_values(inplace=True) s s.min(), s.max()
Note
Note the difference between assigning new categories and reordering the categories: the first renames categories and therefore the individual values in the Series, but if the first position was sorted last, the renamed value will still be sorted last. Reordering means that the way values are sorted is different afterwards, but not that individual values in the Series are changed.
Note
If the Categorical is not ordered, Series.min()
and Series.max()
will raise TypeError
. Numeric operations like +
, -
, *
, /
and operations based on them (e.g. Series.median()
, which would need to compute the mean between two values if the length of an array is even) do not work and raise a TypeError
.
A categorical dtyped column will participate in a multi-column sort in a similar manner to other columns. The ordering of the categorical is determined by the categories
of that column.
python
- dfs = pd.DataFrame({'A' : pd.Categorical(list('bbeebbaa'), categories=['e','a','b'], ordered=True),
'B' : [1,2,1,2,2,1,2,1] })
dfs.sort_values(by=['A', 'B'])
Reordering the categories
changes a future sort.
python
dfs['A'] = dfs['A'].cat.reorder_categories(['a','b','e']) dfs.sort_values(by=['A','B'])
Comparing categorical data with other objects is possible in three cases:
- comparing equality (
==
and!=
) to a list-like object (list, Series, array, ...) of the same length as the categorical data.- all comparisons (
==
,!=
,>
,>=
,<
, and<=
) of categorical data to another categorical Series, whenordered==True
and the categories are the same.- all comparisons of a categorical data to a scalar.
All other comparisons, especially "non-equality" comparisons of two categoricals with different categories or a categorical with any list-like object, will raise a TypeError.
Note
Any "non-equality" comparisons of categorical data with a Series, np.array, list or categorical data with different categories or ordering will raise an TypeError because custom categories ordering could be interpreted in two ways: one with taking into account the ordering and one without.
python
- cat = pd.Series([1,2,3]).astype(
CategoricalDtype([3, 2, 1], ordered=True)
) cat_base = pd.Series([2,2,2]).astype( CategoricalDtype([3, 2, 1], ordered=True) ) cat_base2 = pd.Series([2,2,2]).astype( CategoricalDtype(ordered=True) )
cat cat_base cat_base2
Comparing to a categorical with the same categories and ordering or to a scalar works:
python
cat > cat_base cat > 2
Equality comparisons work with any list-like object of same length and scalars:
python
cat == cat_base cat == np.array([1,2,3]) cat == 2
This doesn't work because the categories are not the same:
python
- try:
cat > cat_base2
- except TypeError as e:
print("TypeError: " + str(e))
If you want to do a "non-equality" comparison of a categorical series with a list-like object which is not categorical data, you need to be explicit and convert the categorical data back to the original values:
python
base = np.array([1,2,3])
- try:
cat > base
- except TypeError as e:
print("TypeError: " + str(e))
np.asarray(cat) > base
When you compare two unordered categoricals with the same categories, the order is not considered:
python
c1 = pd.Categorical(['a', 'b'], categories=['a', 'b'], ordered=False) c2 = pd.Categorical(['a', 'b'], categories=['b', 'a'], ordered=False) c1 == c2
Apart from Series.min()
, Series.max()
and Series.mode()
, the following operations are possible with categorical data:
Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data:
python
s = pd.Series(pd.Categorical(["a","b","c","c"], categories=["c","a","b","d"])) s.value_counts()
Groupby will also show "unused" categories:
python
cats = pd.Categorical(["a","b","b","b","c","c","c"], categories=["a","b","c","d"]) df = pd.DataFrame({"cats":cats,"values":[1,2,2,2,3,4,5]}) df.groupby("cats").mean()
cats2 = pd.Categorical(["a","a","b","b"], categories=["a","b","c"]) df2 = pd.DataFrame({"cats":cats2,"B":["c","d","c","d"], "values":[1,2,3,4]}) df2.groupby(["cats","B"]).mean()
Pivot tables:
python
raw_cat = pd.Categorical(["a","a","b","b"], categories=["a","b","c"]) df = pd.DataFrame({"A":raw_cat,"B":["c","d","c","d"], "values":[1,2,3,4]}) pd.pivot_table(df, values='values', index=['A', 'B'])
The optimized pandas data access methods .loc
, .iloc
, .at
, and .iat
, work as normal. The only difference is the return type (for getting) and that only values already in categories can be assigned.
If the slicing operation returns either a DataFrame or a column of type Series, the category
dtype is preserved.
python
idx = pd.Index(["h","i","j","k","l","m","n",]) cats = pd.Series(["a","b","b","b","c","c","c"], dtype="category", index=idx) values= [1,2,2,2,3,4,5] df = pd.DataFrame({"cats":cats,"values":values}, index=idx) df.iloc[2:4,:] df.iloc[2:4,:].dtypes df.loc["h":"j","cats"] df[df["cats"] == "b"]
An example where the category type is not preserved is if you take one single row: the resulting Series is of dtype object
:
python
# get the complete "h" row as a Series df.loc["h", :]
Returning a single item from categorical data will also return the value, not a categorical of length "1".
python
df.iat[0,0] df["cats"].cat.categories = ["x","y","z"] df.at["h","cats"] # returns a string
Note
This is a difference to R's factor function, where factor(c(1,2,3))[1]
returns a single value factor.
To get a single value Series of type category
pass in a list with a single value:
python
df.loc[["h"],"cats"]
0.17.1
The accessors .dt
and .str
will work if the s.cat.categories
are of an appropriate type:
python
str_s = pd.Series(list('aabb')) str_cat = str_s.astype('category') str_cat str_cat.str.contains("a")
date_s = pd.Series(pd.date_range('1/1/2015', periods=5)) date_cat = date_s.astype('category') date_cat date_cat.dt.day
Note
The returned Series
(or DataFrame
) is of the same type as if you used the .str.<method>
/ .dt.<method>
on a Series
of that type (and not of type category
!).
That means, that the returned values from methods and properties on the accessors of a Series
and the returned values from methods and properties on the accessors of this Series
transformed to one of type category will be equal:
python
ret_s = str_s.str.contains("a") ret_cat = str_cat.str.contains("a") ret_s.dtype == ret_cat.dtype ret_s == ret_cat
Note
The work is done on the categories
and then a new Series
is constructed. This has some performance implication if you have a Series
of type string, where lots of elements are repeated (i.e. the number of unique elements in the Series
is a lot smaller than the length of the Series
). In this case it can be faster to convert the original Series
to one of type category
and use .str.<method>
or .dt.<property>
on that.
Setting values in a categorical column (or Series) works as long as the value is included in the `categories`:
python
idx = pd.Index(["h","i","j","k","l","m","n"]) cats = pd.Categorical(["a","a","a","a","a","a","a"], categories=["a","b"]) values = [1,1,1,1,1,1,1] df = pd.DataFrame({"cats":cats,"values":values}, index=idx)
df.iloc[2:4,:] = [["b",2],["b",2]] df try: df.iloc[2:4,:] = [["c",3],["c",3]] except ValueError as e: print("ValueError: " + str(e))
Setting values by assigning categorical data will also check that the categories match:
python
df.loc["j":"k","cats"] = pd.Categorical(["a","a"], categories=["a","b"]) df try: df.loc["j":"k","cats"] = pd.Categorical(["b","b"], categories=["a","b","c"]) except ValueError as e: print("ValueError: " + str(e))
Assigning a Categorical to parts of a column of other types will use the values:
python
df = pd.DataFrame({"a":[1,1,1,1,1], "b":["a","a","a","a","a"]}) df.loc[1:2,"a"] = pd.Categorical(["b","b"], categories=["a","b"]) df.loc[2:3,"b"] = pd.Categorical(["b","b"], categories=["a","b"]) df df.dtypes
You can concat two DataFrames containing categorical data together, but the categories of these categoricals need to be the same:
python
cat = pd.Series(["a","b"], dtype="category") vals = [1,2] df = pd.DataFrame({"cats":cat, "vals":vals}) res = pd.concat([df,df]) res res.dtypes
In this case the categories are not the same and so an error is raised:
python
df_different = df.copy() df_different["cats"].cat.categories = ["c","d"] try: pd.concat([df,df_different]) except ValueError as e: print("ValueError: " + str(e))
The same applies to df.append(df_different)
.
See also the section on merge dtypes<merging.dtypes>
for notes about preserving merge dtypes and performance.
0.19.0
If you want to combine categoricals that do not necessarily have the same categories, the union_categoricals
function will combine a list-like of categoricals. The new categories will be the union of the categories being combined.
python
from pandas.api.types import union_categoricals a = pd.Categorical(["b", "c"]) b = pd.Categorical(["a", "b"]) union_categoricals([a, b])
By default, the resulting categories will be ordered as they appear in the data. If you want the categories to be lexsorted, use sort_categories=True
argument.
python
union_categoricals([a, b], sort_categories=True)
union_categoricals
also works with the "easy" case of combining two categoricals of the same categories and order information (e.g. what you could also append
for).
python
a = pd.Categorical(["a", "b"], ordered=True) b = pd.Categorical(["a", "b", "a"], ordered=True) union_categoricals([a, b])
The below raises TypeError
because the categories are ordered and not identical.
In [1]: a = pd.Categorical(["a", "b"], ordered=True)
In [2]: b = pd.Categorical(["a", "b", "c"], ordered=True)
In [3]: union_categoricals([a, b])
Out[3]:
TypeError: to union ordered Categoricals, all categories must be the same
0.20.0
Ordered categoricals with different categories or orderings can be combined by using the ignore_ordered=True
argument.
python
a = pd.Categorical(["a", "b", "c"], ordered=True) b = pd.Categorical(["c", "b", "a"], ordered=True) union_categoricals([a, b], ignore_order=True)
union_categoricals
also works with a CategoricalIndex
, or Series
containing categorical data, but note that the resulting array will always be a plain Categorical
python
a = pd.Series(["b", "c"], dtype='category') b = pd.Series(["a", "b"], dtype='category') union_categoricals([a, b])
Note
union_categoricals
may recode the integer codes for categories when combining categoricals. This is likely what you want, but if you are relying on the exact numbering of the categories, be aware.
python
c1 = pd.Categorical(["b", "c"]) c2 = pd.Categorical(["a", "b"])
c1 # "b" is coded to 0 c1.codes
c2 # "b" is coded to 1 c2.codes
c = union_categoricals([c1, c2]) c # "b" is coded to 0 throughout, same as c1, different from c2 c.codes
This section describes concatenations specific to category
dtype. See Concatenating objects<merging.concat>
for general description.
By default, Series
or DataFrame
concatenation which contains the same categories results in category
dtype, otherwise results in object
dtype. Use .astype
or union_categoricals
to get category
result.
python
# same categories s1 = pd.Series(['a', 'b'], dtype='category') s2 = pd.Series(['a', 'b', 'a'], dtype='category') pd.concat([s1, s2])
# different categories s3 = pd.Series(['b', 'c'], dtype='category') pd.concat([s1, s3])
pd.concat([s1, s3]).astype('category') union_categoricals([s1.values, s3.values])
Following table summarizes the results of Categoricals
related concatenations.
arg1 | arg2 | result |
---|---|---|
category | category (identical categories) | category |
category | category (different categories, both not ordered) | object (dtype is inferred) |
category | category (different categories, either one is ordered) | object (dtype is inferred) |
category | not category | object (dtype is inferred) |
You can write data that contains category
dtypes to a HDFStore
. See here <io.hdf5-categorical>
for an example and caveats.
It is also possible to write data to and reading data from Stata format files. See here <io.stata-categorical>
for an example and caveats.
Writing to a CSV file will convert the data, effectively removing any information about the categorical (categories and ordering). So if you read back the CSV file you have to convert the relevant columns back to category and assign the right categories and categories ordering.
python
from pandas.compat import StringIO
python
s = pd.Series(pd.Categorical(['a', 'b', 'b', 'a', 'a', 'd'])) # rename the categories s.cat.categories = ["very good", "good", "bad"] # reorder the categories and add missing categories s = s.cat.set_categories(["very bad", "bad", "medium", "good", "very good"]) df = pd.DataFrame({"cats":s, "vals":[1,2,3,4,5,6]}) csv = StringIO() df.to_csv(csv) df2 = pd.read_csv(StringIO(csv.getvalue())) df2.dtypes df2["cats"] # Redo the category df2["cats"] = df2["cats"].astype("category") df2["cats"].cat.set_categories(["very bad", "bad", "medium", "good", "very good"], inplace=True) df2.dtypes df2["cats"]
The same holds for writing to a SQL database with to_sql
.
pandas primarily uses the value np.nan to represent missing data. It is by default not included in computations. See the Missing Data section
<missing_data>
.
Missing values should not be included in the Categorical's categories
, only in the values
. Instead, it is understood that NaN is different, and is always a possibility. When working with the Categorical's codes
, missing values will always have a code of -1
.
python
s = pd.Series(["a", "b", np.nan, "a"], dtype="category") # only two categories s s.cat.codes
Methods for working with missing data, e.g. ~Series.isna
, ~Series.fillna
, ~Series.dropna
, all work normally:
python
s = pd.Series(["a", "b", np.nan], dtype="category") s pd.isna(s) s.fillna("a")
The following differences to R's factor functions can be observed:
- R's levels are named categories
- R's levels are always of type string, while categories in pandas can be of any dtype.
- It's not possible to specify labels at creation time. Use
s.cat.rename_categories(new_labels)
afterwards. - In contrast to R's factor function, using categorical data as the sole input to create a new categorical series will not remove unused categories but create a new categorical series which is equal to the passed in one!
- R allows for missing values to be included in its levels (pandas' categories). Pandas does not allow NaN categories, but missing values can still be in the values.
The memory usage of a Categorical
is proportional to the number of categories plus the length of the data. In contrast, an object
dtype is a constant times the length of the data.
python
s = pd.Series(['foo','bar']*1000)
# object dtype s.nbytes
# category dtype s.astype('category').nbytes
Note
If the number of categories approaches the length of the data, the Categorical
will use nearly the same or more memory than an equivalent object
dtype representation.
python
s = pd.Series(['foo%04d' % i for i in range(2000)])
# object dtype s.nbytes
# category dtype s.astype('category').nbytes
Currently, categorical data and the underlying Categorical is implemented as a python object and not as a low-level numpy array dtype. This leads to some problems.
numpy itself doesn't know about the new `dtype`:
python
- try:
np.dtype("category")
- except TypeError as e:
print("TypeError: " + str(e))
dtype = pd.Categorical(["a"]).dtype try: np.dtype(dtype) except TypeError as e: print("TypeError: " + str(e))
Dtype comparisons work:
To check if a Series contains Categorical data, use hasattr(s, 'cat')
:
python
hasattr(pd.Series(['a'], dtype='category'), 'cat') hasattr(pd.Series(['a']), 'cat')
Using numpy functions on a Series of type category
should not work as Categoricals are not numeric data (even in the case that .categories
is numeric).
python
s = pd.Series(pd.Categorical([1,2,3,4])) try: np.sum(s) #same with np.log(s),.. except TypeError as e: print("TypeError: " + str(e))
Note
If such a function works, please file a bug at https://github.com/pandas-dev/pandas!
Pandas currently does not preserve the dtype in apply functions: If you apply along rows you get a Series of object
dtype (same as getting a row -> getting one element will return a basic type) and applying along columns will also convert to object.
python
- df = pd.DataFrame({"a":[1,2,3,4],
"b":["a","b","c","d"], "cats":pd.Categorical([1,2,3,2])})
df.apply(lambda row: type(row["cats"]), axis=1) df.apply(lambda col: col.dtype, axis=0)
CategoricalIndex
is a type of index that is useful for supporting indexing with duplicates. This is a container around a Categorical
and allows efficient indexing and storage of an index with a large number of duplicated elements. See the advanced indexing docs <indexing.categoricalindex>
for a more detailed explanation.
Setting the index will create a CategoricalIndex
python
cats = pd.Categorical([1,2,3,4], categories=[4,2,3,1]) strings = ["a","b","c","d"] values = [4,2,3,1] df = pd.DataFrame({"strings":strings, "values":values}, index=cats) df.index # This now sorts by the categories order df.sort_index()
Constructing a Series from a Categorical will not copy the input Categorical. This means that changes to the Series will in most cases change the original `Categorical`:
python
cat = pd.Categorical([1,2,3,10], categories=[1,2,3,4,10]) s = pd.Series(cat, name="cat") cat s.iloc[0:2] = 10 cat df = pd.DataFrame(s) df["cat"].cat.categories = [1,2,3,4,5] cat
Use copy=True
to prevent such a behaviour or simply don't reuse `Categoricals`:
python
cat = pd.Categorical([1,2,3,10], categories=[1,2,3,4,10]) s = pd.Series(cat, name="cat", copy=True) cat s.iloc[0:2] = 10 cat
Note
This also happens in some cases when you supply a numpy array instead of a `Categorical`: using an int array (e.g. np.array([1,2,3,4])
) will exhibit the same behaviour, while using a string array (e.g. np.array(["a","b","c","a"])
) will not.