Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many operators do not work with dtype null #15257

Open
2 tasks done
drhagen opened this issue Mar 23, 2024 · 1 comment
Open
2 tasks done

Many operators do not work with dtype null #15257

drhagen opened this issue Mar 23, 2024 · 1 comment
Labels
bug Something isn't working P-low Priority: low python Related to Python Polars

Comments

@drhagen
Copy link

drhagen commented Mar 23, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
pl.DataFrame({"a": [None, None]}).with_columns(pl.col("a").abs())

Log output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/david/tabeline/.venv/lib/python3.11/site-packages/polars/dataframe/frame.py", line 8366, in with_columns
    return self.lazy().with_columns(*exprs, **named_exprs).collect(_eager=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/tabeline/.venv/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1943, in collect
    return wrap_df(ldf.collect())
                   ^^^^^^^^^^^^^
polars.exceptions.InvalidOperationError: `abs` operation not supported for dtype `null`

Issue description

The following operations cannot be run columns with dtype null:

  • abs
  • sin
  • cos
  • tan
  • arcsin
  • arccos
  • arctan
  • floor
  • ceil
  • is_nan
  • is_finite
  • std
  • var
  • max
  • min
  • sum
  • median
  • any
  • all

All of them except any and all used to complete without error. Issue #14093 seems to suggests that behavior like this is considered a bug.

Expected behavior

Return null for every operation.

Installed versions

--------Version info---------
Polars:               0.20.16
Index type:           UInt32
Platform:             Linux-5.15.0-101-generic-x86_64-with-glibc2.31
Python:               3.11.3 | packaged by conda-forge | (main, Apr  6 2023, 08:57:19) [GCC 11.3.0]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               <not installed>
pyarrow:              <not installed>
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>
@drhagen drhagen added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Mar 23, 2024
@deanm0000 deanm0000 added P-low Priority: low and removed needs triage Awaiting prioritization by a maintainer labels Mar 24, 2024
@Elvynzs
Copy link

Elvynzs commented Mar 28, 2024

I have encountered the same issue in my usage of polars.

This result in some extra code to handle pl.Null inputs as an edge case. (In my use case, I do not know in advance the dtype of the input)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P-low Priority: low python Related to Python Polars
Projects
Status: Ready
Development

No branches or pull requests

3 participants