Documentation for "SINGLE FILE DATA STORAGE OPTIONS" Missing

[obsidience]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

Hi all,

The following page shows a tip link at the bottom for converting the dataset to a single file. Upon navigating to the "result-storage" page, I don't see any information on how to achieve this.

https://crawlee.dev/docs/introduction/saving-data#whats-datasetpushdata

Thanks!

Code sample

No response

Package version

3.10.2

Node.js version

20.13.1

Operating system

Win11

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[obsidience]

As a side note, how do I combine all pushData calls into a single file for debugging purposes?

[B4nan]

the link is correct, it should point to the KVS section, the sentence in question literally says "see the Result storage guide for Key-value stores"

anyway, sounds like you want crawler.exportData() instead, that's a newer helper that will export the whole dataset into a single CSV or JSON file

https://crawlee.dev/api/basic-crawler/class/BasicCrawler#exportData

or there is also this example if you want to work with the dataset directly and not the crawler object

https://crawlee.dev/docs/examples/export-entire-dataset

[obsidience]

Thanks for your help B4nan, I can be dense at times (not joking!) but where in the "Key-value store" section does it indicate how to "store your data in a single big file"?

The last link appears to be what I was after, thank you!

[B4nan]

with KVS you store data into a single file, thats the whole point of KVS - you store a value (your data, all of them), under a specific key. unlike dataset which is append only, you can modify the value stored in KVS.

[obsidience]

Thanks, still learning... By dataset being "append only", do you mean that pushData() is always writing to a new file? In programming jargon, the Push/pop term usually infers changes to an existing collection. So, I had assumed that multiple calls to "await Dataset.pushData({foo: "foo"});" would be additive to an existing or newly created object array that can later be stored as a single file.