Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

If url goes to file, dont try to fetch markup #1

Closed
hmol opened this issue Feb 5, 2016 · 1 comment
Closed

If url goes to file, dont try to fetch markup #1

hmol opened this issue Feb 5, 2016 · 1 comment

Comments

@hmol
Copy link
Owner

hmol commented Feb 5, 2016

When crawling http://www.the-website-to-crawl.com and the application gets to a url for a file; http://www.the-website-to-crawl.com/reports/report.pdf , it will throw exception. This is beacuse it will then try to fetch html markup from pdf-file. So, when getting to a url for file, dont fetch markup, just continue.
@hmol hmol closed this as completed Feb 9, 2016
@hmol
Copy link
Owner Author

hmol commented Feb 9, 2016

Fixed

hmol pushed a commit that referenced this issue Feb 19, 2017
@tdwright
Merge pull request #1 from hmol/develop
1942d54 
Merging changes from hmol/Linkcrawler
robsiera pushed a commit to robsiera/LinkCrawler that referenced this issue Oct 30, 2020
@ed-graham
Merge pull request hmol#1 from ed-graham/feature/show-new-links-for-3…
5ca2f87 
…xx-responses

Print Location header for 3xx and also any exception messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hmol