feat: add FileDownload "crawler"

[barjin]

Adds a new package @crawlee/file-download, which overrides the HttpCrawler's MIME type limitations and allows the users to download arbitrary files.

Aside from the regular requestHandler, this crawler introduces streamHandler, which passes a ReadableStream with the downloaded data to the user handler.

[B4nan]

Do we really need a new package here? If there are no additional dependencies, I would just expose the new class in the HTTP crawler package.

[barjin]

Yup, I don't have any strong opinion here - the (very short) discussion was at https://github.com/apify/store-website-content-crawler/issues/242

[barjin]

The latest commit adds streamHandler, which is mutually exclusive with the requestHandler in the constructor and allows the users to work with the data stream instead of the fully downloaded data.

const crawler = new FileDownload({
    async streamHandler({ stream }) {
        const file = createWriteStream('test.webm');
        pipeline(stream, file, (err) => {
            if (err) {
                console.error('Pipeline failed', err);
            }
        });
    },
})

This is a direct port from WCC. What do you think, is it worth it to keep it here? Or should we rely on the requestHandler only?

[B4nan]

Can we add some tests for this?

I dont mind having the stream handler.

[janbuchar]

This is actually pretty impressive! Could you also add an example to the docs? Or a whole page even...

[janbuchar]

Could we just... not support this in new crawlers?

[barjin]

Why so? The last time I checked (last week), the Delivery team loves a good RequestList (for whatever reason).

Plus, this behaviour is coming from BasicCrawler (snippet here), so all our crawlers are doing this right now. I'm not here to gatekeep this, just curious about your motivation to remove this :)

[B4nan]

for whatever reason

The cost is indeed a valid reason, since with RL there is no cost for the RQ writes (and they can be indeed costly for a high number of requests, e.g. in this run the RQ writes are about 70% of the cost). I will recheck this with RL in place to see what is the actual cost difference.

[janbuchar]

Well, it's a piece of non-trivial logic that I thought was not really necessary for this use case. And in general, less features means less code to maintain 🤷

[barjin]

was not really necessary for this use case

Right, but it's free (inherited from BasicCrawler). I copied (and edited) the JSDoc from HttpCrawler, so it may be too descriptive in some places, happy to get some feedback for that too. This part is mentioned in all the crawlers we have, though.

[B4nan]

The cost is indeed a valid reason, since with RL there is no cost for the RQ writes (and they can be indeed costly for a high number of requests, e.g. in this run the RQ writes are about 70% of the cost). I will recheck this with RL in place to see what is the actual cost difference.

Btw here is the same actor, but with RL in place. The cost is ~5$ while with RQ it's 26$, this is surely a good enough reason to keep this abstraction (unfortunately, lol).