Skip to content

Allow distinct timeouts for differently labeled handlers in the router #1485

@damianr13

Description

@damianr13
Contributor

Describe the feature
I want to be able to specify different timeouts for handlers with different labels. If I expect the website I am crawling to have two or more types of pages, I am doing different things on each of those types, so the time it takes to process a page is different.

Motivation
I am trying to crawl a category page with "infinite scroll" + "load more button" instead of pagination. Similarly to the example in the tutorial (https://crawlee.dev/docs/introduction/scraping), I have 2 types of pages: LIST and DETAIL.

Currently I am facing the problem of hitting a timeout before being able to load all the elements on the LIST page. I looked it up and I found the parameter requestHandlerTimeoutSecs that could be passed to the crawler to increase the timeout limit. My understanding is that this limit applies to all the requests regardless of their types, but I would still like to keep the limit for an individual DETAIL page lower than the high timeout value I need to specify for the LIST page.

Activity

added
featureIssues that represent new features or improvements to existing features.
on Aug 25, 2022
mnmkng

mnmkng commented on Aug 25, 2022

@mnmkng
Member

Thanks for the feature suggestion @damianr13

@B4nan we should do this. I literally had the same idea 1 hour ago when playing with the router middlewares. I even think we should allow the router to set different preNavigationHooks and other options.

self-assigned this
on Sep 12, 2022
self-assigned this
on Sep 21, 2022
removed their assignment
on Sep 21, 2022
added
t-toolingIssues with this label are in the ownership of the tooling team.
on Sep 12, 2023
added this to the 4.0 milestone on Jul 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureIssues that represent new features or improvements to existing features.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Relationships

None yet

    Participants

    @mtrunkat@B4nan@damianr13@mnmkng@szmarczak

    Issue actions

      Allow distinct timeouts for differently labeled handlers in the router · Issue #1485 · apify/crawlee