Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove jinja2 blocks from robots.txt #5648

Merged
merged 1 commit into from
Oct 6, 2020

Conversation

smotornyuk
Copy link
Member

@smotornyuk smotornyuk commented Oct 6, 2020

Previously, robots.txt was rendered as a Jinja2 template and had few blocks for extensions. Now it served as a static file and doesn't need those blocks, because an extension can just contain its own robots.txt in the public folder, which will be served instead

@amercader amercader merged commit cc000d3 into ckan:master Oct 6, 2020
nickumia-reisys added a commit to GSA/ckanext-datagovcatalog that referenced this pull request Aug 14, 2021
…c file; the new functionality needs review, ckan/ckan#5648; all tests passing now
@mtearle
Copy link

mtearle commented Oct 21, 2021

What happens when multiple extensions need to update the robots.txt file?

@smotornyuk
Copy link
Member Author

Extension, that registers its public folder first, wins.

If you know you are using multiple extensions that redefine robots.txt, you have to create an application-specific extension and merge inside it all the required robots.txt from the different extensions.
And don't forget to put your application-specific extension at the beginning of the plugins list, so that it registers its public folder first.

@mtearle
Copy link

mtearle commented Oct 22, 2021

That implies that the person writing and and the person installing the extension needs to be aware of this behaviour, rather than just using the templating engine to handle it.

For example, I might write an authentication extension and want to append to robots.txt to keep robots away from my URL paths, and I also have another extension doing my theming/HTML customizations and also have reasons to modify the robots.txt there. My authentication extension was excellent and others now wish to adopt it but I now have to put a warning in my documentation to note that my extension may screw up the robots.txt for your site.

Long story short, I'm struggling to see what the use case/bug was that required removing the customization/templating from robots.txt?

@Zharktas
Copy link
Member

It was done as part of python3 migrations 8ee2148 for simplicity sake. However it can be reimplemented as flask view if someone has the time for it.

@mtearle
Copy link

mtearle commented Oct 22, 2021

Thanks. That makes sense :) I will note it as an issue to be tackled for the migration of our site off 2.8.3

@mtearle
Copy link

mtearle commented Jul 12, 2022

#6930 notes the issue, with a pull request offering a fix in #6953

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants