Develop an example using the HTTP client as a web crawler #108

glynos opened this Issue Apr 25, 2012 · 2 comments


None yet
2 participants

glynos commented Apr 25, 2012

A good example of the HTTP client would be as a web crawler. This could also be a good demonstration of the flexibility of the URI class.


deanberris commented Apr 25, 2012

This sounds like a good idea. However I'm afraid of the effort that would need to go into something like this. Parsing HTML is scary stuff and finding the URL's and interpreting rel="nofollow" in a DOM along with the myriad other link elements in a document is a tall order.

That said, I'd be willing to accept contributions to this effect.


glynos commented Apr 26, 2012

My motivation for opening this issue is to start generating more complex and interesting examples for the HTTP client. At this stage I am not worried about the complexity of the HTML parsing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment