Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Instruct crawlers to avoid indexing tools pages #1077

Open
ahayes opened this issue Aug 24, 2022 · 0 comments
Open

Instruct crawlers to avoid indexing tools pages #1077

ahayes opened this issue Aug 24, 2022 · 0 comments
Assignees
@NassefAsh
Labels
Enhancement Desired improvement to design or implementation that adds value. Java Nunaliit Java related
Milestone
2.2.9

Comments

@ahayes
Copy link
Member

ahayes commented Aug 24, 2022

Nunaliit dynamically serves robots.txt content pointing to sitemap.xml content. Somewhere in there, we should tell crawlers to avoid indexing /tools/*. Google seems to get grumpy about Soft 404 Errors when a page like the submission tool is accessed anonymously and shows some errors as a result.
@ahayes ahayes added Enhancement Desired improvement to design or implementation that adds value. Java Nunaliit Java related labels Aug 24, 2022
@ahayes ahayes added this to the 2.2.9 milestone Aug 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NassefAsh NassefAsh

Labels
Enhancement Desired improvement to design or implementation that adds value. Java Nunaliit Java related
Projects
None yet
Milestone
2.2.9
Development

No branches or pull requests
2 participants
@ahayes @NassefAsh