Adding code to detect and report broken links.

[snehm]

Current Issue: Broken links for which server returns 404 status (or similar error) are not reported as failed links, instead these erroneous pages are parsed for text content.

[s0md3v]

Thanks for this one but instead of raising an exception, Photon should append the url to the failed set.
Can you please do it that way?

[snehm]

Hi, Thank you for your reply.
These exceptions are handled by the very next except block (inside same function), which adds the url to the failled set and continue normal execution for other links. Is there any problem with this approach?

[s0md3v]

The failed set doesn't actually contain the URLs with a 4xx status, they are just simply the URLs that returned None because that's how an exception will occur.
I agree that catching non-exsistent pages is a good idea but you are simply comparing the status code with 200 which is surely a bad idea because the whole range i.e. 200-299 is dedicated to successful status.
On top of that, raising an exception is useless.

[s0md3v]

I want to fix this asap but I am not sure if it would be a good thing to do it myself instead of waiting for you to fix your code 😕

[snehm]

Hey, I am doing it now =D

[s0md3v]

You may want to do this:

>>> response = requests.get('http://example.com/good_page.php')
>>> response.status_code
200
>>> response.status_code == requests.codes.ok
True
>>> response = requests.get('http://example.com/bad_page.php')
>>> response.status_code
404
>>> response.status_code == requests.codes.ok
False

[snehm]

Hi, Please review changes :)

[s0md3v]

Just tested this against https://rocket.chat and got some false positives.
Not sure what is causing them, I need some time to troubleshoot, prolly 15 minutes.

[s0md3v]

Tip 💡 : Always use 4 spaces or a tab of equivalent width for indentation, that's the standard. You were using 3, I had to fix it.