New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot apply lambda function to DataFrame with columns of type numpy.datetime64 #18573

Closed
meditativeape opened this Issue Nov 30, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@meditativeape

meditativeape commented Nov 30, 2017

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

ts1 = np.datetime64('2017-11-29T03:30')
ts2 = np.datetime64('2017-11-29T03:45')
d = {'number' : pd.Series([1., 2.]), 'string': pd.Series(['haha', 'wawa']), 'datetime64': pd.Series([ts1, ts2])}
df = pd.DataFrame(d)
df.apply(lambda row: (row.number, row.string), axis=1)

Problem description

Applying lambda function to a DataFrame with columns of type numpy.datetime64 throws an unexpected error: ValueError: Shape of passed values is (2, 2), indices imply (2, 3). The error message is not helpful, and I believe this error is thrown due to a bug. Dropping columns of type numpy.datetime64 fixes the issue.

Expected Output

0    (1.0, haha)
1    (2.0, wawa)
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.14.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 5.4.1
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

@jreback jreback added this to the No action milestone Nov 30, 2017

@jreback

This comment has been minimized.

Contributor

jreback commented Nov 30, 2017

duplicate of #15628

.apply has to guess the return shape from your opaque passed function. This is not always possible.

idiomatically what you are doing is quite odd, simply do this.

In [43]: df[['number', 'string']]
Out[43]: 
   number string
0     1.0   haha
1     2.0   wawa
@meditativeape

This comment has been minimized.

meditativeape commented Nov 30, 2017

Thanks for the suggestion, @jreback !

jreback added a commit to jreback/pandas that referenced this issue Dec 2, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 3, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 7, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 10, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 14, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 21, 2017

jreback added a commit to jreback/pandas that referenced this issue Dec 23, 2017

jreback added a commit to jreback/pandas that referenced this issue Jan 6, 2018

API/BUG: .apply will correctly infer output shape when axis=1
closes #16353
closes #17348
closes #17437
closes #18573
closes #17970
closes #17892
closes #17602
closes #18775
closes #18901
closes #18919

jreback added a commit to jreback/pandas that referenced this issue Feb 5, 2018

API/BUG: .apply will correctly infer output shape when axis=1
closes #16353
closes #17348
closes #17437
closes #18573
closes #17970
closes #17892
closes #17602
closes #18775
closes #18901
closes #18919

jorisvandenbossche added a commit that referenced this issue Feb 7, 2018

harisbal pushed a commit to harisbal/pandas that referenced this issue Feb 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment