-
Notifications
You must be signed in to change notification settings - Fork 620
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: ignore invalid URLs in enqueueLinks
in browser crawlers
#1803
Conversation
Invalid URL
errorInvalid URL
error
Invalid URL
errorInvalid URL
error
Invalid URL
errorInvalid URL
error
@mnmkng @vladfrangu @B4nan could anybody take a look at this? I'm happy with making code changes if needed. |
Thanks for the PR. I'm not sure if we want to silently ignore invalid URLs. Maybe this could be an option of I would prefer not to ignore invalid URLs, because it usually points to some issues or inconsistencies in the crawlers. This way, we could be missing data and not even knowing about it. |
@B4nan At least instead of just crashing with |
I agree it should not be silently ignored. I can add a new option |
Sounds good to me! Btw it also feels weird that your PR changes only the browser crawler, we should ensure the other crawlers work the same - this should be probably implemented on lower level. |
Other crawlers use the crawlee/packages/cheerio-crawler/src/internals/cheerio-crawler.ts Lines 228 to 232 in cf64729
jsdom-crawler: crawlee/packages/jsdom-crawler/src/internals/jsdom-crawler.ts Lines 268 to 273 in cf64729
So if we would add the new |
Right, I thought we do this somewhere already. In that case, we should align the behavior probably. |
…r when encountering invalid URLs
I aligned the |
Invalid URL
errorenqueueLinks
in browser crawlers
Fixes: #1802