Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SchemaError for Non-Exiting dtype on Concat #17300

Open
2 tasks done
starzar opened this issue Jun 30, 2024 · 3 comments
Open
2 tasks done

SchemaError for Non-Exiting dtype on Concat #17300

starzar opened this issue Jun 30, 2024 · 3 comments
Labels
bug Something isn't working needs repro Bug does not yet have a reproducible example python Related to Python Polars

Comments

@starzar
Copy link

starzar commented Jun 30, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

dfAnalysis_total = pl.DataFrame()
df_1 = 18-06-2024(CotFinancials)_dfAnalysis_cur
df_2 = 25-06-2024(CotFinancials)_dfAnalysis_cur
df_list =  [df_1,df_2]

for df_cur in range(df_list):
    # SchemaError: type; Int64 is incompatible with expected type Float64
    # But I64 does not exist in any of the concatenated dataframes
    dfAnalysis_total = pl.concat([dfAnalysis_total,df_cur],how="diagonal_relaxed")

Log output

Traceback (most recent call last):
  File "C:\Users\User_0\Documents\Code\Python\NseScraping\Others\CotReport\py\cotAnalysis_financials.py", line 219, in txt_to_df
    cotSoloAnalysis(cot_df_total.with_row_index(),fileName)
  File "C:\Users\User_0\Documents\Code\Python\NseScraping\Others\CotReport\py\cotAnalysis_financials.py", line 437, in cotSoloAnalysis
    dfAnalysis_total = pl.concat(items = [dfAnalysis_total,dfAnalysis_cur],how="diagonal")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User_0\AppData\Local\Programs\Python\Python312\Lib\site-packages\polars\functions\eager.py", line 196, in concat
    out = wrap_df(plr.concat_df_diagonal(elems))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.SchemaError: type Int64 is incompatible with expected type Float64

Issue description

https://drive.google.com/file/d/1x182gU8SwwcuHtXGom8JsGC1kuIUC_Br/view?usp=sharing
I64 does not exist in any of the concatenated dataframes but polars gives " SchemaError: type; Int64 is incompatible with expected type Float64"

Expected behavior

I64 should not shown as a dtype at all.

Installed versions

polars 0.20.31
@starzar starzar added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Jun 30, 2024
@ritchie46
Copy link
Member

Can you create a reproducible example?

@ritchie46 ritchie46 added incomplete Incomplete issue: needs MWE needs repro Bug does not yet have a reproducible example and removed needs triage Awaiting prioritization by a maintainer incomplete Incomplete issue: needs MWE labels Jun 30, 2024
@starzar
Copy link
Author

starzar commented Jul 1, 2024

Error starts when concatenating 2nd row onwards -
shape: (1, 5)
┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐
│ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │
╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡
│ {{5637.0,10503.0, ┆ {{761.0,1541.0,14 ┆ {{813.0,22040.0,2 ┆ {{2763.0,13974.0, ┆ {{690.0,1812.0,3 │
│ 761.0},{11.3,21.0 ┆ 59.0},{1.5,3.1,2. ┆ 6343.0},{1.6,44.0 ┆ 6392.0},{5.5,27.9 ┆ 06.0},{1.4,3.6,0 │
│ ,1.5},{166.0,-470 ┆ 9},{761.0,-434.0, ┆ 99998,52.700001}, ┆ ,12.8},{89.0,774. ┆ .6},{159.0,-179. │
│ 4.0,761.0},{4.0,5 ┆ 655.0},{15.0,11.0 ┆ {221.0,3443.0,739 ┆ 0,382.0},{0.0,0.0 ┆ 0,37.0},{0.0,0.0 │
│ .0,15.0}} ┆ ,7.0}} ┆ 9.0},{97.0,25.0,8 ┆ ,0.0}} ┆ ,0.0}} │
│ ┆ ┆ .0}} ┆ ┆ │
└───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘
cot_df_cur
shape: (1, 5)
┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐
│ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │
╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡
│ {{0.0,3701.0,0.0} ┆ {{0.0,0.0,0.0},{0 ┆ {{0.0,902.0,17450 ┆ {{215.0,21676.0,3 ┆ {{0.0,2517.0,310 │
│ ,{0.0,14.6,0.0},{ ┆ .0,0.0,0.0},{0.0, ┆ .0},{0.0,3.6,68.9 ┆ 634.0},{0.8,85.59 ┆ .0},{0.0,9.9,1.2 │
│ 0.0,1084.0,0.0},{ ┆ 0.0,0.0},{0.0,0.0 ┆ 00002},{0.0,723.0 ┆ 9998,14.4},{184.0 ┆ },{0.0,172.0,-2. │
│ 0.0,0.0,0.0}} ┆ ,0.0}} ┆ ,2855.0},{7.0,4.0 ┆ ,1879.0,-1163.0}, ┆ 0},{0.0,0.0,0.0} │
│ ┆ ┆ ,123.0}} ┆ {19.0,0.0,0.0}} ┆ } │
└───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘
cot_df_curtype Int64 is incompatible with expected type Float64

shape: (1, 5)
┌───────────────────┬───────────────────┬───────────────────┬───────────────────┬──────────────────┐
│ Dealer ┆ AssetManager ┆ LeveragedFunds ┆ OtherReportables ┆ NonReportables │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] ┆ struct[4] │
╞═══════════════════╪═══════════════════╪═══════════════════╪═══════════════════╪══════════════════╡
│ {{127.0,22378.0,0 ┆ {{0.0,13428.0,464 ┆ {{529.0,17397.0,1 ┆ {{444.0,4651.0,17 ┆ {{1.0,3691.0,174 │
│ .0},{0.3,55.59999 ┆ .0},{0.0,33.29999 ┆ 2994.0},{1.3,43.2 ┆ 15.0},{1.1,11.6,4 ┆ 2.0},{0.0,9.2,4. │
│ 8,0.0},{127.0,208 ┆ 9,1.2},{-872.0,-2 ┆ 00001,32.299999}, ┆ .3},{-914.0,2697. ┆ 3},{-28.0,-170.0 │
│ 91.0,-872.0},{0.0 ┆ 99.0,69.0},{41.0, ┆ {-99.0,1470.0,-16 ┆ 0,-686.0},{0.0,0. ┆ ,-448.0},{0.0,0. │
│ ,22.0,41.0}} ┆ 11.0,14.0}} ┆ 001.0},{9.0,0.0,0 ┆ 0,0.0}} ┆ 0,0.0}} │
│ ┆ ┆ .0}} ┆ ┆ │
└───────────────────┴───────────────────┴───────────────────┴───────────────────┴──────────────────┘
type Int64 is incompatible with expected type Float64

@cmdlineluser
Copy link
Contributor

@starzar You'll need to provide a full code example in order for other people to run it and reproduce the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs repro Bug does not yet have a reproducible example python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

3 participants