Best practices designing and scaling scraping flow

[noam-cohen]

Hi

Background

I am working on building a scraping flow.
We want to use prefect however we are having some issues.
We are using the following DAG:

with Flow('scraping_flow') as flow:   
    url_collection = extract_urls()
    url_transformed = transform_url.map(url_collection)

    # depth first execution mode
    fetched_res = fetch_request.map(url=url_transformed, rm=unmapped(request_manager))
    status_writer.map(data=fetched_res, key=url_transformed, status=CrawlerStates.FETCHED)
    parsed_res = parse_response.map(url=url_collection, response=fetched_res, scraper=unmapped(crunchbase_scraper))
    entitized_result = entitize_result.map(parsed_res, entitizer=unmapped(entity_transformer))
    results = load_result.map(entitized_result)

Open questions

The things that we are not sure about-

1. Can we scale the process for millions of URLs, while prioritizing depth over breadth (finishing to store the scraped data before moving to the next URL)? From a short test I've did the DFE takes to much time to trigger the second level and actually works breath first.
2. How can we fetch in batches s.t. we won't need to store in memory millions of URLs when starting the flow?
3. Is there a different design that we should use instead? Is prefect the right tool here?