Is there a way to see, which page was parrent for an URL we are crawling?

[huglester]

Hello,

For example I do:
$urls = (new Centipede\Crawler('domain.com'))->crawl();

and then try to Guzzle get all the pages to check for statuses, etc.

But for example I see, that url '/en/abou-us' is not accessible. (gives 404). Is there a way, to check, on which of parent URLs this '/en/about-us' was found? Because it could be that the link is inside 1 article, or so. sot it would be hard hunting.

Thank you!

[umpirsky]

@huglester No, but I guess you can override Centipede\Crawler and build a tree of urls.