Nxtscape agent does not respect or request robots.txt

[Xe]

Nxtscape is making automated requests and thus needs to follow the Robots Exclusion Protocol. Failing that, it must clearly identify itself in the User-Agent or some other header when it is making requests that are not a human directly clicking on links.

[elliots]

Strong disagree. The agent is acting “as” the user, in their authentication context, and on behalf of them and their desire. It is a user agent in the same way as any other browser.

Also, in your request “needs” and “must” I think should be replaced with “could” or “I think should”. There’s no law here.

(Not part of this project, just my opinion).

[jcpsimmons]

Xe Iaso (@Xe) , I am disappointed, which is why I put the confused face emoji on your original post. I am a longtime admirer of your blog, software development work, and other writings where you consistently treat complex issues with the nuance and grace that is generally lacking from online texts these days.

I know you know that the first prerequisite to determining course on this matter is the ontological problem of "what is a robot?" (which is insufficiently defined here) This problem has two subproblems:

1. Where does this particular application fall on the robot <-> not robot continuum`?
2. Who has the power to define the terms of this continuum.

2 is the bigger problem at hand, already obliquely stated by Elliot Shepherd (@elliots) i.e.

There's no law here.

If one calls upon the mighty auspices of the venerable World Wide Web Consortium* for guidance in this matter, one will find no directive that we are commanded to abide by. I would be happy to be wrong about this in which case we could swiftly put the matter to rest but I'm unable to find any such literature.

* Even then, W3C makes generally good calls most of the time but they don't have unilateral legal authority over the web.

All that aside, this looks like some cool passion project by just a few engineers (not a huge company) so does it really matter?