Feature Request: Add more I/O functions to the DuckDB PySpark API & toPandas

[TomBurdge]

Why do you want this feature?

The DuckDB PySpark API is a brilliant idea and it doesn't yet cover some core parts of PySpark's API.

Some useful attributes to add to the experimental PySpark DataFrame class are:

• .write.parquet
• .write.csv
• .toPandas

I have implemented them in my own fork.
This was very simple because the necessary functions already exist in the Spark relation attribute (duckdb.DuckDBPyRelation).

I will make a pull request if the core team thinks this is a good idea!
This would be my first issue & pull request.

I think the use case is here for DuckDB PySpark and I am in the process of also doing a short writeup which shows how to switch the IOs between DuckDB spark and PySpark when using dagster.
Having these features in would make the code a little simpler for the writeup (it's not too hard right to make the right dagster IO class currently - you just need to use the .relation attribute, which isn't part of the spark API).

[Tishj]

Those sound like great additions 👍

[TomBurdge]

Awesome! thanks @Tishj. I haven't written the new unit tests yet. Once I have, I will make a pull request.

[TomBurdge]

@Tishj I have now made a pull request for this feature here.
Some more future easy wins might be rank, dense_rank and DataFrameWriter.json method.