Web Archive (WARC) Support

[Ristellise]

Which package is the feature request for? If unsure which one to select, leave blank

None

Feature

Crawlee supports other export formats such as json but doesn't seem to have any support for Web Archive formats.

Motivation

Saving it as web archive enables developers to psot-process the data as much as as the crawler has gathered.

Ideal solution or implementation, and any additional constraints

Optional feature switch integration with fastwarc library is preferred. Crawlee should be able to write raw file contents into the fastwarc library and fastwarc should be able to provide for the rest.

Alternative solutions or implementations

Alternatively, warcio is an alternative library for consideration. However, that is slower in python as it's written in pure python compared to fastwarc which is written mainly in C.

Other context

Was introduced to crawlee at pyconsg 2025

[barjin]

Hello, and thank you for your interest in this project!

You actually can write a WARC-file-creating crawler yourself with Crawlee already. Consider the following snippet:

import { CheerioCrawler } from '@crawlee/cheerio';
import { WARCRecord, WARCSerializer } from "warcio";
import { createWriteStream } from "fs";

async function main() {
    const warcArchive = createWriteStream("archive.warc");

    const serializedWARCInfo = await WARCSerializer.serialize(await WARCRecord.createWARCInfo(
        {
            filename: "archive.warc",
            warcVersion: "WARC/1.1"
        },
        {}
    ));
    warcArchive.write(serializedWARCInfo);

    const crawler = new CheerioCrawler({
        requestHandler: async ({ request, response, body, enqueueLinks }) => {
            console.log(`Processing ${request.url}...`);

            async function* content() {
                yield new TextEncoder().encode(body as string);
            }

            const record = await WARCRecord.create(
                {
                    url: response.url,
                    type: 'response',
                    httpHeaders: response.headers as any,
                },
                content(),
            );

            warcArchive.write(await WARCSerializer.serialize(record));

            await enqueueLinks();
        },
        maxRequestsPerCrawl: 10,
    });

    await crawler.run(['https://crawlee.dev']);
    warcArchive.end();
}

main();

You see that you can e.g., write a new record to the WARC file stream for each site Crawlee visits.

---

Regarding the libraries you mentioned, this repository contains the TypeScript version of Crawlee. For Crawlee for Python, go to https://github.com/apify/crawlee-python/. The Crawlee for Python maintainers have already opened an issue here (apify/crawlee-python#1262). We'd be grateful if you could share your use case and the interface you'd like to see for this feature.

I'll keep this issue open to track parity w/ apify/crawlee-python#1262 .

[Ristellise]

Thanks for the reply! I didn't know that this is for the typescript version.

I currently wrote a custom webscraper that uses playwright directly to emulate a browser session (with adblock capabilities via a chrome extension that I hacked in with a extension and additional JavaScript spoofing for bypassing some elements).

Current use case... Well I'd think that example would kind of have satisfied my needs. The only problem that I can think of is duplication of page / url content. (Not sure if it's possible other than keeping a cache.)

My current headaches with dealing with my own code (ala own library) are:

1. For whatever odd reason, the initial seed url get's sent twice. Likely due to extension doing a reload.
2. No built-in deduplication at scale (I think this should be solve separately with some sort of database to keep track of what has been crawled)

I'm considering using crawlee as it appears to be more battletested and saves me the time having the wrangle with concurrency and scalability within python. :)