I'd recommend to use parquet partitions

[veonua]

it gives the smaller size and faster save\load time, while supported by the majority of data libraries

[veonua]

wallstreetbets_posts.csv > 920Mb
wallstreetbets_posts.parquet ~ 120Mb

[mattpodolak]

Hi @veonua, thanks for pointing this out. However, this is not functionality of the pmaw library, but post-processing done after responses have been retrieved with pmaw so I will be closing this issue.

The documentation will continue to provide an example using .csv, as this benefits the largest number of users.

[veonua]

the proposal was for a cache as well

[mattpodolak]

ah okay, sorry, that wasn't clear. I know it takes up less space than .pickle which is currently being used, but I haven't done any benchmarks with it yet.

Added as a feature request - #7. Thanks!