Add Polars #1171
Description
Please consider adding Polars to kaggle docker image.
Polars is a DataFrames library that can be used with Python. It is really fast, supports lazy execution and provides an alternative API to pandas. It solves lots of issues I face when using Kaggle notebooks. For context, I work mostly on tabular and time series data.
Pandas is slow when it comes to group by, which is quite frequently needed for feature engineering. Polars on the other hand uses multithreaded lock-free approach. It turns out to be very very fast to summarize data and join different data frames. I don't mind to do feature engineering on my own laptop at all, but when it comes to submission and debug submission errors, the lack of performance is pretty painful. Benchmarks (H2O, TPCH) show it has at least 10x speedup over Pandas and Dask.
Poloars supports lazy execution, so when it comes to a large csv file, it doesn't need to read everything into memory. The output can be fetched when you finish the query and ask Polars to fetch it. I can't stress this enough because Kaggle notebook's memory capacity is quiet restricted which pretty much makes me give up preprocessing data on Kaggle notebooks.
I work with Pandas almost everyday so I'm quite used to the API. But after writing probably over 100 weird groupby().apply(lambda sth) operations which are not even performant (I sometimes have to write a multiprocessing groupby myself, eww), I believe providing an aternative API to python users on Kaggle is a good idea. This post shows a good example.
Polars's repo has 6.5k stars and it has 120+ comments on Hacker News. I think there should be enough evidence to show that the library is popular and in great need for data scientists on Kaggle.
I don't mind sending a PR if this isn't going to be too cubersum. Happy to discuss further, cheers!