Parameter to ignore robots.txt

[rodolphopivetta]

Congratulations by the project, this is awesome.

I'm having a specific use case with this project: I have a staging web project that I don't want it to be indexed by the google. In other words, my robots.txt have one rule to ignore all links, and in this case, I can't use the linkcheck to check that.

Would be great have a parameter that says I don't want to honors the robots.txt.

[filiph]

Thanks @rodolphopivetta! The parameter you're asking for would be useful, but also extremely risky, imho. There are bad people out there who would love to have an extremely fast, easy to use web spider that ignores netiquette.

For link-checking on your own sites (especially in staging), you can append your robots.txt to allow linkcheck.

User-agent: linkcheck
Disallow:

Hope this helps. I'm closing the issue but feel free to reopen if I'm completely off the mar.

[rodolphopivetta]

Thank you @filiph, this will work for me, but just to keep note, the correct robots.txt that I've used was:

User-agent: linkcheck
Disallow:

User-agent: *
Disallow: /

[filiph]

Thanks! I updated my answer above in order to not confuse folks.