.transform inconsistent / error-prone behavior for list #28246
Labels
Apply
Apply, Aggregate, Transform
Bug
Groupby
Nested Data
Data where the values are collections (lists, sets, dicts, objects, etc.).
Code Sample, a copy-pastable example if possible
Problem description
I was hinted that it could be a bug: https://stackoverflow.com/questions/57743798/pandas-transform-inconsistent-behavior-for-list
There are two things, but let focus on 1 first:
Result of
df['n_list_problem'] = df.groupby(['label'])[['wave']].transform(list)
is not consistent with similar operations likedf['n_tuple'], df['n_set']
etc.On stackoverflow I was pointed to issues regarding series lenght for rationale, but consensus was that, this behavior is confusing and error-prone.
The 2 problem:
df['n_sum'] = df.groupby(['label'])['wave'].transform(sum)
works as expected, butdf['n_tuple_problem'] = df.groupby(['label'])['wave'].transform(tuple)
gives unexpected result.To get proper one, you need to coerce series into dataframe by
[[]]
instead of[]
forwave
.df['n_tuple'] = df.groupby(['label'])[['wave']].transform(tuple)
Expected Output
df['n_list_expected'] = df.groupby(['label'])[['wave']].transform(tuple)['wave'].apply(list)
Output of
pd.show_versions()
INSTALLED VERSIONS
commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-141-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 0.25.1
numpy : 1.17.0
pytz : 2019.2
dateutil : 2.8.0
pip : 19.1.1
setuptools : 41.0.1
Cython : 0.29.13
pytest : None
hypothesis : None
sphinx : 2.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.7.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
The text was updated successfully, but these errors were encountered: