-
-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix regressions with failOnFailedSeed
option
#572
Conversation
When pages are unreachable due to DNS not resolving, the crawler was failing with exit code 17, which conflicts with expected and documented behavior.
This prevents a breaking change from the 0.x series of releases, which did not have failOnInvalidStatus, which may otherwise cause unexpected results for users on upgrading.
Hmm, I think that may be more confusing. The --failOnInvalidStatus was introduced in 1.x, and yes it is a breaking change from 0.x, but I think that's ok, given it was a major version change. I think there should be a way to only fail on invalid/unloadable pages, and pages that do load, but return an error code. The browser does not treat 4xx/5xx responses as failures, as they do load, and #563 is referring to invalid/unloadable URLs specifically it seems. |
@ikreymer Reverted and tests extended to include status code, which let me catch another issue in the argParser. Should be all good now! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the cli updates to make it more clear
Fixes #563
This PR makes a few changes to fix a regression in behavior around
failOnFailedSeed
for the 1.x releases:failOnFailedSeed
is setfailOnINvalidStatus
isn't set(Edited after reverting change related to invalid statuses)