Is there a way to export prediction SplinkDataFrame to parquet directly without going through Pandas?

[bingbong-sempai]

Hi! Just wondering if it is possible to export the predictions table directly to parquet without going through pandas?
It seems to be possible through duckdb but i'm not sure how a SplinkDataFrame is accessed that way.

[RobinL]

Yes, you can use df_predict.as_spark_dataframe()

splink/splink/spark/linker.py

Line 63 in c8e093a

def as_spark_dataframe(self):

[bingbong-sempai]

Apologies, I forgot to mention I'm using the duckdb backend.
I think I found it!
linker._con.sql(f"COPY {predictions.physical_name} TO outputs/predictions.parquet")