Skip to content
This repository has been archived by the owner on Dec 19, 2023. It is now read-only.

Account for 418s in crawling #66

Open
capjamesg opened this issue Sep 16, 2022 · 0 comments
Open

Account for 418s in crawling #66

capjamesg opened this issue Sep 16, 2022 · 0 comments
Assignees
@capjamesg
Labels
crawler enhancement New feature or request

Comments

@capjamesg
Copy link
Owner

IndieWeb Search should account for 418s. If one is encountered, the crawler should wait the Retry-After period. If that period is greater than one day, another request should be made. If it happens three times in a row, the crawl will be suspended.
@capjamesg capjamesg self-assigned this Sep 16, 2022
@capjamesg capjamesg added enhancement New feature or request crawler labels Sep 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@capjamesg capjamesg

Labels
crawler enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@capjamesg