feat!: update built-in Rust Polars to 2023-04-20

[eitsupi]

I have not yet been able to correct the errors by breaking changes.

If anyone wants to work on this, you can push it to this branch.

The upstream includes various breaking changes.
Two in particular that affect many areas are

• perf(rust, python): use inlined strings for field and schema polars#7272 (Introduce smartstring)
• feat(rust, python): The big error revamp polars#7362 (Change the structure of error messages)

TODO

• Add more tests for new options
  • dt_replace_time_zone's use_earliest
  • str_parse_int's strict
• Update NEWS about new features and breaking changes

[sorhawell]

Sometimes in the past when bumping rust-polars, it can become quite messy with 10-20 hotfixes here and there. The breaking changes are fortunately easy to catch when compiling. Non breaking changes can be hard to notices and update the code base for. Maybe polars supports some new join strategy, but r-polars will not allow before someone files an issue.

[sorhawell]

I fixed all compiler errors. Still missing: update tests + evaluate rust-polars changes and file some issues.

[eitsupi]

Remain test failures

• Error (test-expr_arr.R:4:3): arr$lengths

  > pl$DataFrame(list_of_strs = pl$Series(list(list(),NULL)))
  Error: in Series.new: SchemaMismatch(ErrString("invalid series dtype: expected `List`, got `f64`"))

• Error (test-expr_arr.R:396:3): concat

  > df = pl$DataFrame(x = "a", y = "b")
  > df$select(pl$col("x")$arr$concat(pl$col("y")))
  Error: in Series.new: InvalidOperation(ErrString("new series from rtype ExternalPtr is not supported (yet)"))

• Error (test-expr_arr.R:454:3): eval
  Seems the same error of Error (test-expr_arr.R:396:3): concat from concat_list
• Failure (test-expr.R:1512:3): hash + reinterpret
• Failure (test-expr.R:2176:3): extend_expr
• Error (test-expr.R:2339:3): concat_list
  Seems the same error of Error (test-expr_arr.R:396:3): concat from concat_list

[sorhawell]

@eitsupi oh more edits, flag me when ready :)

[eitsupi]

I would like to eventually version up to the latest commit at this time. Thoughts?

[sorhawell]

I'm puzzled about how the "NA" / NA conversion error started. I fixed it in this PR 9d93a08-9e6137f , is not related to bumping rust-polars.
It is now also an issue in join_asof PR. It must be some new crate or rextendr triggering it. The issue is extendr has an known tricky as_str() method which will convert R NA_character into &str "NA", which is not very desirable.

I will delay with updating other PRs until this fix has been merged.

[eitsupi]

I'm puzzled about how the "NA" / NA conversion error started. I fixed it in this PR 9e6137f , is not related to bumping rust-polars.

As I commented on #190 (comment), I think this is due to the new release of waldo.

[sorhawell]

As I commented on #190 (comment), I think this is due to the new release of waldo.

Nice then I have had a bug in these conversion forever, and now waldo sees it.

[sorhawell]

I would like to eventually version up to the latest commit at this time. Thoughts?

like bumping to polars 0.6.0? I aggree

oh you mean merge in main? I aggree also :)

maybe bumping polars can be delayed to after this merge

[eitsupi]

maybe bumping polars can be delayed to after this merge

Indeed, I was hoping to upgrade to today's version since rust-polars is currently 2 weeks old in this PR, but given that there are bugs that will be fixed by this PR, it seems better to merge now.

[eitsupi]

I think I have completed all the work and can merge if there are no problems with CI.

[eitsupi]

Can I merge this?

[sorhawell]

Can I merge this?

yeah

[eitsupi]

Thanks all!

[vincentarelbundock]

This is very cool! I think there are some new methods in the latest version. When I find time, I'll try to make a list and see which ones I'm able to implement.