Some feature questions

[koolma]

Hi,
I have some questions regarding the features of this crawler, which are not covered by the documentation.

1. Does php-spider support JavaScript (content and URLs generated via JavaScript)
2. Does php-spider follows robot.txt files?
3. Is php-spider able to leverage a sitemap?
4. Is it possible to crawl sites that require authentication?

[danvuquoc]

@koolma, I'm not a contributor or maintainer of this project but I have used this spider.

1. It doesn't support javascript, this isn't a headless browser. It simply uses guzzle http 6 to grab pages. Typically you then use a css/xpath discovery class to find new uris to find more pages to crawl.
2. It doesn't follow robots.txt files, but you could easily write a filter that implements the PreFetchFilterInterface and uses tomverran/robots to match the uri path about to be grabbed by the spider against the robots.txt. I've done this as well with robots headers and robots meta tags implementing a PostFetchFilterInterface.

I'm not sure about 3 and 4, hope this helps.

[solverat]

3. no, not without any additional work
4. yes, just implement some middleware

[mvdbos]

Good answers by @danvuquoc and @solverat. Closing this ancient issue. :-)