Upgrade PyO3 to 0.21 and switch to new API

[itamarst]

Description

In PyO3 0.21, &'py PyAny will be replaced by Bound<'py, PyAny>. The benefits for upgrading are increased performance, the costs of not upgrading are that the old API is deprecated and will, eventually, go away: https://polar.sh/davidhewitt/posts/replacing-pyo3-api-pt1

https://pyo3.rs/latest/migration documents the upgrade process (at time of writing 0.21 is still beta, but will presumably be released soon). Based on their suggestions, I think this would look like this for Polars:

1. First PR: Switch to 0.21, enable gil-refs feature, fix other migration issues.
2. A series of PRs that migrate over to Bound. The development process would involve disabling gil-refs temporarily, locally only, to spot APIs that need migration, but given this would probably be a series of PRs, the feature should continue to be on in main.
3. Final PR: When no deprecation warnings are left, disable gil-refs.

• Initial PyO3 0.21 migration, with gil-refs feature enabled #15621
• Part 3 PyO3 0.21 upgrade: Remove usage of deprecated numpy crate APIs #15666
• Part 4 PyO3 0.21 upgrade: Switch over to Bound APIs in the pyo3 crate #15669
• More updates to PyO3 0.21 Bound<> APIs #15785
• Even more updates to PyO3 0.21 Bound<> APIs #15864
• More PyO3 0.21 Bound<> APIs #15890
• Continue the very long journey towards PyO3 0.21 Bound<> APIs #16078
• Continue PyO3 0.21 Bound<> updates, and remove "gil-refs" pyo3 crate feature #16142
• Update documentation to talk about Bound<> APIs instead of deprecated PyO3 APIs #16155
• Switch to PyO3 0.21's native datetime support, now available with abi3 #16158

Followups include:

• Upgrade pyo3-polars and friends to PyO3 0.21 and its new Bound<> APIs pyo3-polars#81
• Finish switching to PyO3 0.21's native datetime support, now available with abi3, once Python 3.9 is the minimum version #16199

[itamarst]

This is something I can help work on.

[stinodego]

Yes we will definitely upgrade to the new PyO3 version when it's released. I had planned to work on that myself but if it's something you're interested in working on I think it's fine if you want to do the honors here.

[itamarst]

0.21.1 is out, so I will start on this either this week or next.

[stinodego]

Something about support for chrono? Need to get reminder from @stinodego

PyO3 has support for the Python datetime module:
https://pyo3.rs/main/doc/pyo3/types/struct.pydatetime

However, this was never available under the abi3 feature flag.

So we have a bunch of workarounds in place to materialize/read datetime objects. Specially, we have defined Python functions here:
https://github.com/pola-rs/polars/blob/main/py-polars/polars/_utils/convert.py

...that are called from Rust, for example here:

polars/py-polars/src/conversion/any_value.rs

Lines 61 to 64 in 8ef2e21

	AnyValue::Date(v) => {
	let convert = utils.getattr(intern!(py, "to_py_date")).unwrap();
	convert.call1((v,)).unwrap().into_py(py)
	},

The 0.21 release notes mention:

Extended chrono / datetime conversions, including support for the abi3 feature and the chrono-tz crate

So I am expecting that we can now take advantage of native PyO3 datetime support. And I expect a good performance boost + code cleanup to come from this.

[itamarst]

I'm gonna say this is done, insofar as it covers the main Polars repository, and the pyo3-polars issue is already filed for that repository.

[stinodego]

Thanks a lot for the thorough work @itamarst !