Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disallow forms (with CAPTCHA) to bots #438

Merged
merged 2 commits into from
Jun 14, 2024

Conversation

chenejac
Copy link
Contributor

@chenejac chenejac commented Jan 18, 2024

VIVO GitHub issue: 3935

Linked VIVO PR

What does this pull request do?

Disallow access to /contact and /forgot-password to bots (at least to bots which respect robots.txt)

What's new?

robots.txt is updated

How should this be tested?

Follow the instructions in Linked VIVO PR

Interested parties

Tag (@ mention) interested parties or, if unsure, @VIVO-project/vivo-committers

Copy link
Member

@ivanmrsulja ivanmrsulja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@ivanmrsulja ivanmrsulja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small change is requested as we changed this from kebbab-case to camel-case in the meantime.:

image

The other rules work as intended (I tested them using Merkle):

Screenshot from 2024-05-31 12-15-41
Screenshot from 2024-05-31 14-23-08

webapp/src/main/webapp/robots.txt Outdated Show resolved Hide resolved
Co-authored-by: Ivan R. Mršulja <nighteliteace@gmail.com>
Copy link
Member

@ivanmrsulja ivanmrsulja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have re-run all the tests using Merkle and now everything works as intended. Steps to reproduce the tests:

  • Setup a publicly available Vitro server (I reccommend using a tool like NGROK)
  • If the vivo does not run on the root URL and instead you have to go to /vitro or something similar, you have to provide robots.txt manually in the text editor
  • Choose the crawler of your choice from dropdown menu and try to fetch any of the disallowed domains

@litvinovg litvinovg merged commit c532266 into vivo-project:main Jun 14, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants