Skip to content

Unify RequestList and RequestProvider interfaces and extract their "tandem" behavior from BasicCrawler #2499

@janbuchar

Description

@janbuchar
Contributor

Initial steps have been taken in #2498.

  • Introduce a IRequestProvider interface
    Unify IRequestProvider with IRequestList
    Introduce a "switching" request provider implementation to better encapsulate the logic of "read from list first, add new requests to queue" currently present in BasicCrawler
    The transfer from request list to request queue should be done in background, not when fetching requests from the queue

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Participants

      @janbuchar

      Issue actions

        Unify `RequestList` and `RequestProvider` interfaces and extract their "tandem" behavior from `BasicCrawler` · Issue #2499 · apify/crawlee