ScottMansfield / widow Public

Notifications
Fork 4
Star 24

Code
Issues 12
Pull requests
Actions
Projects
Wiki
Security
Insights

Code
Issues
Pull requests
Actions
Projects
Wiki
Security
Insights

Issues: ScottMansfield/widow

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

12 Open 7 Closed

Author

Filter by author

Label

Filter by label

Use alt + click/return to exclude labels

or ⇧ + click/return for logical OR

Projects

Filter by project

Milestones

Filter by milestone

Assignee

Filter by who’s assigned

Assigned to nobody

Sort

Sort by

Newest Oldest Most commented Least commented Recently updated Least recently updated Best match

Most reactions

Issues list

Throttle requests to a domain by total bandwidth in a specified period of time enhancement fetch parse

#19 opened Sep 22, 2015 by ScottMansfield

All outbound requests should have a User-Agent attached to them enhancement fetch Minor parse

#18 opened Jun 18, 2015 by ScottMansfield

Check for robots meta tag while parsing a page enhancement Medium parse

#15 opened May 26, 2015 by ScottMansfield

Add support for @import in CSS files and inline <style> contents bug enhancement parse

#13 opened May 26, 2015 by ScottMansfield

mailto:, phone:, etc break parsing bug enhancement Major parse

#11 opened May 19, 2015 by ScottMansfield

Implment rate-limiting on a per-host basis enhancement fetch Major

#10 opened May 12, 2015 by ScottMansfield

Add links by content type to the main page data enhancement parse

#9 opened May 12, 2015 by ScottMansfield

Have a better story around local caching independent of the crawling stages enhancement fetch index Major parse

#6 opened May 8, 2015 by ScottMansfield

Investigate more accurate timing of website response enhancement fetch Minor

#5 opened May 8, 2015 by ScottMansfield

Support for sitemap.xml fetch Major parse

#4 opened May 8, 2015 by ScottMansfield

Add support for If-Modified-Since and ETag headers enhancement fetch Major parse

#3 opened May 8, 2015 by ScottMansfield

Add support for robots.txt for any website bug enhancement fetch Major parse

#2 opened May 8, 2015 by ScottMansfield

ProTip! Adding no:label will show everything without a label.