Bump rust-polars to 0.40.0

[etiennebacher]

https://github.com/pola-rs/polars/releases/tag/rs-0.40.0

@eitsupi I'll have limited time in the next few days to address this (we are not in rush, it's just so you know)

[eitsupi]

@eitsupi I'll have limited time in the next few days to address this (we are not in rush, it's just so you know)

Thanks for starting the work. Of course I'll take a look at this, but it looks like a pretty tough update and I don't know if I can complete it in a few days.

[eitsupi]

It can now be compiled, but the rolling functions will probably need to be rewritten.

[eitsupi]

@etiennebacher Thanks for your great works!

On my machine, the R session hangs on the expr_name test (so I have renamed the test file to disable the tests for now) and the following part seems to cause a panic.

r-polars/tests/testthat/test-series.R

Line 508 in 3d2a333

ul = pl$DataFrame(s)$select(pl$col("")$flatten()$flatten()$flatten())$to_list() |>

I think the hang in the R session may be due to the implementation of Send and Sync at the following location, but I do not know why this has now occurred.
I also do not know why @sorhawell implemented this.

r-polars/src/rust/src/utils/extendr_concurrent.rs

Lines 11 to 18 in 3d2a333

	//shamelessly make Robj send + sync
	//no crashes so far for the 'data'-SEXPS as Vectors, lists, pairlists
	//mainly for debug should not be used in production
	//CLOSEXP crashes every time
	#[derive(Clone, Debug)]
	pub struct ParRObj(pub Robj);
	unsafe impl Send for ParRObj {}
	unsafe impl Sync for ParRObj {}

[eitsupi]

Hmmm, it looks like there is a problem with the way the DataFrame is constructed.
It panics when I try to create a DataFrame with columns of nested List types, which is not a problem in Python.

> pl$DataFrame(a = as_polars_series(list(list(1, 2, 3), list(4, 5, 6))))
thread '<unnamed>' panicked at /usr/local/cargo/git/checkouts/polars-b0d90607192fd414/7bc7014/crates/polars-core/src/series/mod.rs:149:37:
called `Result::unwrap()` on an `Err` value: InvalidOperation(ErrString("`vec_hash` operation not supported for dtype `list[f64]`"))
thread '<unnamed>' panicked at src/rdataframe/mod.rs:89:1:
explicit panic
Error: Execution halted with the following contexts
   0: In R: in $DataFrame():
   0: During function call [pl$DataFrame(a = as_polars_series(list(list(1, 2, 3), list(4, 
          5, 6))))]
   1: user function panicked: select

> pl$select(a = as_polars_series(list(list(1, 2, 3), list(4, 5, 6))))
thread '<unnamed>' panicked at /usr/local/cargo/git/checkouts/polars-b0d90607192fd414/7bc7014/crates/polars-core/src/series/mod.rs:149:37:
called `Result::unwrap()` on an `Err` value: InvalidOperation(ErrString("`vec_hash` operation not supported for dtype `list[f64]`"))
thread '<unnamed>' panicked at src/rdataframe/mod.rs:89:1:
explicit panic
Error in .pr$DataFrame$select(self, unpack_list(..., .context = "in $select()")) : 
  user function panicked: select

[eitsupi]

There are several unresolved issues with this PR, but I think we can merge and do the follow-up work for now.

[eitsupi]

map method errors are only on Linux, so may be related to pola-rs/polars#16365
I thought I had made modifications, but they may not have been sufficient.

Edit: It does not seem related.

I take a look the remain issue.

[eitsupi]

More related issue: pola-rs/polars#12568

[eitsupi]

I do not know why $name$map() does not work only on Ubuntu.
I would like to vote to remove the function as it seems almost unnecessary to me.

[eitsupi]

I don't understand why this process has to be done on the Rust side.
Wouldn't it be enough to receive the function as an argument in rename, process it on the R side, and then pass it as a list to Rust's rename?
(tidypolars should work well doing that with dplyr::rename_with().)
https://github.com/pola-rs/polars/blob/17cf8b7e17974b51c9bdc760422005fe591bc7c5/py-polars/polars/lazyframe/frame.py#L4393-L4445

In any case, I think we can merge them as they are now and fix them later. (I don't think I will have enough time for a while down the road)

[eitsupi]

This is a side note, but what I have learned is that upstream is not very interested in Rust's breaking changes, and the release notes are of little use.

So in the future, I think it is better to proactively incorporate changes from HEAD without waiting for Rust releases.

[etiennebacher]

Thanks a lot, I'll take a look at the remaining issues to see if I can come up with a fix for them. In any case, I'll update the NEWS before merging.

This is a side note, but what I have learned is that upstream is not very interested in Rust's breaking changes, and the release notes are of little use.

The release notes are sometimes useful. They are messy but it can really help going through bugs or changes in implementation. I agree we shouldn't feel forced to wait for an official release but I also think it matches quite well the pace of development (release every ~1.5 month).

There were several trivial bug fixes upstream just after 0.40.0 were released, I think we should bump the version to an unreleased version but we should do it in another PR.

[eitsupi]

Follow-up tasks should include the followings:

• Update polars to newer edge version.
• Add *_by() methods.
• Fix pl$select() for nested list types.
• Remove $name$map() and add a functionality rename with R function to $rename() (or adds $rename_with())