polars filter-with $mask won't work

[maxim-uvarov]

Describe the bug

@ayax79 I compiled nushell and polars from the latest main (including commit 9836998)
However polars won't accept the example from the book, that was previously working (see below)

How to reproduce

let new = [9 8 4] | polars into-df
let mask = $new == 8
$new | polars filter-with $mask

Error: nu::shell::cant_convert

  × Can't convert to NuExpression.
   ╭─[entry #3:1:27]
 1 │ $new | polars filter-with $mask
   ·                           ──┬──
   ·                             ╰── can't convert NuDataFrameCustomValue to NuExpression
   ╰────

Expected behavior

I expect polars to filter the $new df

Screenshots

No response

Configuration

key	value
version	0.93.1
major	0
minor	93
patch	1
branch	main
commit_hash	c54d223
build_os	macos-aarch64
build_target	aarch64-apple-darwin
rust_version	rustc 1.77.2 (25ef9e3d8 2024-04-09)
rust_channel	1.77.2-aarch64-apple-darwin
cargo_version	cargo 1.77.2 (e52e36006 2024-03-26)
build_time	2024-05-07 05:50:32 +00:00
build_rust_channel	release
allocator	mimalloc
features	dataframe, default, sqlite, system-clipboard, trash, which
installed_plugins	explore, image, inc, polars

Additional context

No response

[fdncred]

No issues for me.

❯ let new = [9 8 4] | polars into-df
❯ $new 
╭─#─┬─0─╮
│ 0 │ 9 │
│ 1 │ 8 │
│ 2 │ 4 │
╰───┴───╯
❯ let mask = $new == 8
❯ $mask 
╭─#─┬───0───╮
│ 0 │ false │
│ 1 │ true  │
│ 2 │ false │
╰───┴───────╯
❯ $new | polars filter-with $mask 
╭─#─┬─0─╮
│ 0 │ 8 │
╰───┴───╯

Built yesterday

key	value
version	0.93.1
major	0
minor	93
patch	1
branch	main
commit_hash	c70c43a
build_os	linux-x86_64
build_target	x86_64-unknown-linux-gnu
rust_version	rustc 1.77.2 (25ef9e3d8 2024-04-09)
rust_channel	1.77.2-x86_64-unknown-linux-gnu
cargo_version	cargo 1.77.2 (e52e36006 2024-03-26)
build_time	2024-05-13 07:05:06 -05:00
build_rust_channel	release
allocator	mimalloc
features	dataframe, default, sqlite, system-clipboard, trash, which
installed_plugins	bg, emoji, explore, file, formats, from_parquet, gstat, inc, json_path, jwalk, md, msgpack, pnet, polars, query, query_git, regex, str_similarity, stress_internals

[maxim-uvarov]

Thank you for checking, @fdncred

I'm sorry, I might have messed up with something, but I checked and recompiled everything for more than 3 times and I'm still getting the error. Maybe it is mac related :(

> git pull
> cargo install --path . --features=dataframe
> nu
> cd crates/nu_plugin_polars
> cargo install --path .
> plugin add '/Users/user/.cargo/bin/nu_plugin_polars'
> nu
> let new = [9 8 4] | polars into-df
∙ let mask = $new == 8
∙ $new | polars filter-with $mask
Error: nu::shell::cant_convert

  × Can't convert to NuExpression.
   ╭─[entry #1:3:27]
 2 │ let mask = $new == 8
 3 │ $new | polars filter-with $mask
   ·                           ──┬──
   ·                             ╰── can't convert NuDataFrameCustomValue to NuExpression
   ╰────

key	value
version	0.93.1
major	0
minor	93
patch	1
branch	main
commit_hash	c54d223
build_os	macos-aarch64
build_target	aarch64-apple-darwin
rust_version	rustc 1.77.2 (25ef9e3d8 2024-04-09)
rust_channel	1.77.2-aarch64-apple-darwin
cargo_version	cargo 1.77.2 (e52e36006 2024-03-26)
build_time	2024-05-07 05:50:32 +00:00
build_rust_channel	release
allocator	mimalloc
features	dataframe, default, sqlite, system-clipboard, trash, which
installed_plugins	explore, image, inc, polars

[fdncred]

Let me recompile the plugin + nushell again and see.

[maxim-uvarov]

It seems like I made a mistake with git, my nushell git commit is still old. c54d223
I'm sorry, trying to find out the solution

[fdncred]

oops, you're right. The latest version of the plugin doesn't work for this example. The older version of the plugin I was running did work. Not sure what version that was.

❯ let new = [9 8 4] | polars into-df
❯ let mask = $new == 8
❯ $new | polars filter-with $mask
Error: nu::shell::cant_convert

  × Can't convert to NuExpression.
   ╭─[entry #3:1:27]
 1 │ $new | polars filter-with $mask
   ·                           ──┬──
   ·                             ╰── can't convert NuDataFrameCustomValue to NuExpression
   ╰────

[maxim-uvarov]

oops, you're right. The latest version of the plugin doesn't work for this example. The older version of the plugin I was running did work. Not sure what version that was.

I checked this example on the nu_plugin_polars version 0.93.0, and it definitely worked back then. I think this bug might be related to the lazyframes refactoring.

[devyn]

I had a look at the source code and it looks like filter-with has been changed to be the same as filter, i.e. there is no code for mask handling now, which is probably incorrect