Skip to content

Is there some feature to make Crawlee continue crawl from the stop place? #2058

Answered by B4nan
qkxie asked this question in Q&A
Discussion options

You must be logged in to vote

The crawler state should be still there (in the storage folder) and should be respected next time you run the crawler, all you need to do is disable the auto-purging of the state, e.g. via CRAWLEE_PURGE_ON_START env var, e.g. CRAWLEE_PURGE_ON_START=0 npm start.

Replies: 2 comments 3 replies

Comment options

You must be logged in to vote
3 replies
@qkxie
Comment options

@Yash-946
Comment options

@ylwu
Comment options

Answer selected by qkxie
Comment options

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
5 participants