Why Spark produces performance data based on csv dataset

[Tom-Deng]

The performance of Spark based on the csv dataset is not optimal, but the current performance data of Spark is based on the csv dataset. What is the author's consideration?

[jangorecki]

Do you mean that Spark is not dealing well with CSV data?
It loads the data into memory so CSV is just a source for data. Later on spark operates on an in-memory object and not the CSV.

[Tom-Deng]

Do you mean that Spark is not dealing well with CSV data? It loads the data into memory so CSV is just a source for data. Later on spark operates on an in-memory object and not the CSV.
Spark has specifically optimized parquet files. Take q1 of 5G data groupby operation as an example，spark processing csv dataset takes 8.42s on average. but spark processing parquet dataset takes 555ms on average.
csv data:

parquet data:

[jangorecki]

Thanks for investigation. You should try to cache data in memory before running first query, as we do in benchmark script. As of now your code is probably including time to load data from csv in the time of query.

[Tom-Deng]

Thanks for investigation. You should try to cache data in memory before running first query, as we do in benchmark script. As of now your code is probably including time to load data from csv in the time of query.
Refer to the suggestions above，I try to cache data in memory before running first query，but the result is basically the same.

[jangorecki]

I don't think that your code caches data in memory. Spark interface is lazy and AFAIR calling persist alone does not force computation.

[Tom-Deng]

I don't think that your code caches data in memory. Spark interface is lazy and AFAIR calling persist alone does not force computation.

See my code in the attachment, you can take a look at it as a whole.
spark_csv_vs_parquet.ipynb.zip

[jangorecki]

It may be easier if you just run db-benchmark script having data source replaced and compare timings.

[Tom-Deng]

It may be easier if you just run db-benchmark script having data source replaced and compare timings.
Caching data in memory，the performance of csv and parquet is indeed similar in db-benchmark's scense.