Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pd.concat dataframes with different datetime64 types #53640

Closed
2 of 3 tasks
Charlie-XIAO opened this issue Jun 13, 2023 · 1 comment · Fixed by #53641
Closed
2 of 3 tasks

BUG: pd.concat dataframes with different datetime64 types #53640

Charlie-XIAO opened this issue Jun 13, 2023 · 1 comment · Fixed by #53641
Assignees
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Milestone

Comments

@Charlie-XIAO
Copy link
Contributor

Charlie-XIAO commented Jun 13, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> df1 = pd.DataFrame({"a": range(2), "b": range(2)}, dtype="datetime64[s]")
>>> df2 = pd.DataFrame({"a": range(2), "b": range(2)}, dtype="datetime64[ns]")
>>> pd.concat([df1, df2])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\reshape\concat.py", line 390, in concat
    return op.get_result()
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\reshape\concat.py", line 678, in get_result
    new_data = concatenate_managers(
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\internals\concat.py", line 198, in concatenate_managers
    return BlockManager(tuple(blocks), axes)
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\internals\managers.py", line 988, in __init__
    self._verify_integrity()
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\internals\managers.py", line 995, in _verify_integrity
    raise_construction_error(tot_items, block.shape[1:], self.axes)
  File "D:\Academic\Open Source\pandas-yxiao\pandas\core\internals\managers.py", line 2218, in raise_construction_error
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (2, 2), indices imply (4, 2)

Issue Description

Note that this bug does not exist on the latest version of pandas (v2.0.2). I think axis may have been messed up somewhere, will investigate.

Expected Behavior

>>> import pandas as pd
>>> df1 = pd.DataFrame({"a": range(2), "b": range(2)}, dtype="datetime64[s]")
>>> df2 = pd.DataFrame({"a": range(2), "b": range(2)}, dtype="datetime64[ns]")
>>> pd.concat([df1, df2])
                            a                           b
0         1970-01-01 00:00:00         1970-01-01 00:00:00
1         1970-01-01 00:00:01         1970-01-01 00:00:01
0         1970-01-01 00:00:00         1970-01-01 00:00:00
1  1970-01-01 00:00:00.001000  1970-01-01 00:00:00.001000

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 4149f323882a55a2fe57415ccc190ea2aaba869b
python           : 3.9.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.22621
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : Chinese (Simplified)_China.936

pandas           : 2.1.0.dev0+956.g4149f32388
numpy            : 1.24.3
pytz             : 2023.3
dateutil         : 2.8.2
setuptools       : 56.0.0
pip              : 23.1.2
Cython           : 0.29.33
pytest           : 7.3.1
hypothesis       : 6.75.9
sphinx           : 6.2.1
blosc            : 1.11.1
feather          : None
xlsxwriter       : 3.1.2
lxml.etree       : 4.9.2
html5lib         : 1.1
pymysql          : 1.0.3
psycopg2         : 2.9.6
jinja2           : 3.1.2
IPython          : 8.14.0
pandas_datareader: None
bs4              : 4.12.2
bottleneck       : 1.3.7
brotli           :
fastparquet      : 2023.4.0
fsspec           : 2023.5.0
gcsfs            : 2023.5.0
matplotlib       : 3.7.1
numba            : 0.57.0
numexpr          : 2.8.4
odfpy            : None
openpyxl         : 3.1.2
pandas_gbq       : None
pyarrow          : 12.0.0
pyreadstat       : 1.2.2
pyxlsb           : 1.0.10
s3fs             : 2023.5.0
scipy            : 1.10.1
snappy           :
sqlalchemy       : 2.0.15
tables           : 3.8.0
tabulate         : 0.9.0
xarray           : 2023.5.0
xlrd             : 2.0.1
zstandard        : 0.21.0
tzdata           : 2023.3
qtpy             : None
pyqt5            : None
@Charlie-XIAO Charlie-XIAO added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 13, 2023
@topper-123 topper-123 added Reshaping Concat, Merge/Join, Stack/Unstack, Explode and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 13, 2023
@Charlie-XIAO
Copy link
Contributor Author

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants