Disallow forms (with CAPTCHA) to bots

[chenejac]

VIVO GitHub issue: 3935

Linked Vitro PR

What does this pull request do?

Disallow access to /contact and /forgot-password to bots (at least to bots which respect robots.txt)

What's new?

robots.txt is updated

How should this be tested?

Run VIVO and try to access to /contact and /forgotPassword from the web browser (this should work), and then testing robots.txt file by using some validator such as this one. Please note that you run VIVO at some public address as a root application (meaning it should not be http://somedomain.com/vivo, it should be http://somedomain.com)

Interested parties

Tag (@ mention) interested parties or, if unsure, @VIVO-project/vivo-committers

[wwelling]

https://github.com/search?q=repo%3Avivo-project%2FVitro%20%2Fcontact&type=code

https://github.com/search?q=repo%3Avivo-project%2FVitro+%2FsubmitFeedback+&type=code

https://github.com/vivo-project/Vitro/pull/421/files#diff-a461e0c1d25d9f1433d95156cd7e1b2a79f90d138b2d46c95889567c2fb742d7R39

[ivanmrsulja]

Works as intended. But keep in mind that the instructions in robots.txt files cannot enforce crawler behavior to the site, just suggest it 😄 to stop crawlers from accessing the pages we can try to manually whitelist user agents on the backend but I think it is not needed.

[milospp]

Works as intended. But keep in mind that the instructions in robots.txt files cannot enforce crawler behavior to the site, just suggest it 😄 to stop crawlers from accessing the pages we can try to manually whitelist user agents on the backend but I think it is not needed.

I don't think we should manually detect robots and disable those pages, because we cannot be 100% sure who is robot and who is not just by header. That's why we have captcha on those pages.

[ivanmrsulja]

One change requested as in: vivo-project/Vitro#438 (review)

[ivanmrsulja]

I have re-run all the tests using Merkle and now everything works as intended. Steps to reproduce the tests:

• Setup a publicly available VIVO server (I reccommend using a tool like NGROK)
• If the vivo does not run on the root URL and instead you have to go to /vivo or something similar, you have to provide robots.txt manually in the text editor
• Choose the crawler of your choice from dropdown menu and try to fetch any of the disallowed domains

[balmas]

This works as advertised. Tested with Merkle and confirmed that submitFeedback, contact and forgotPassword are all disallowed.