extraction strategies are not async

[ederuiter]

crawl4ai version

2025-feb-alpha-1

Expected Behavior

With most of the codebase having async variants; I was suprised to see that the extraction is not async. I expected it to be.

Current Behavior

Extracting can take up longer (especially with LLMExtraction) than the scraping part. As this is currently not async (just parallelized with a thread executor) it will block any async functions until this is completed.

So I would expect something like this to happen

• scrape url1
• start extracting data from url1
• scrape url2
• start extracting data from url2
• completed extracting data from url1
• completed extracting data from url2

Instead this happens

• scrape url1
• start extracting data from url1
• completed extracting data from url1
• scrape url2
• start extracting data from url2
• completed extracting data from url2

Almost as it is not async at all.

Is this reproducible?

Yes

Inputs Causing the Bug

any

Steps to Reproduce

1. use llmextraction (as this can take long, it shows the problem faster)
2. start scraping multiple urls with `arun_many`
3. observe that the second url will only start with the scraping steps after the first has finished extracting.

Code snippets

OS

any

Python version

any

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[ederuiter]

The draft PR is ready; I don't think this is the ultimate solution as this could create a large amount of concurrent requests .. I think this needs something like a semaphore in there that limits the overall outstanding requests.

But before investing more time into this I wanted to check if something like this would be considered.

[aravindkarnam]

@ederuiter It's a brilliant suggestion. But this seems more like a feature request instead of bug. I tried to convert this to our discussions forum, so we could continue the conversation there, but i think GH is blocking me from doing that. Perhaps due to the linked PR.

Anyway we want to reserve GH issues for bugs and use discussions for these conversations.

So I'm closing the issue here (the idea is still brilliant and valid, but it's not a bug). Feel free to start a discussion in https://github.com/unclecode/crawl4ai/discussions/categories/feature-requests. We can properly plan this and align it with the future roadmap of the product.

You can also join our discord channel here https://discord.gg/wpYFACrHR4, for a more realtime conversation.

[aravindkarnam]

Also the closing of the PR was by mistake. Once the changes in the base branch got merged, I deleted it, but that cascaded into deletion of all open PRs on it. It's totally by accident, Now I learned! From next time I'll keep my stale branches around for a few months until I move all the open PRs on it. Anyways, see you in discord!