Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FEATURE: remove topic rss from robots
Crawlers love hitting the rss feeds (confirmed that both Google and Bing do) Experimenting with the impact of blocking these feeds and forcing Crawlers to hit the content direct. It is better if they hit the actual page to start with as opposed to 1. Hit RSS feed 2. Find new content 3. Hit post link 4. Get canonical 5. Hit canonical Lots of pointless work. We do not know for sure what impact this will have on newsreader apps, we will listen for feedback.
- Loading branch information
f40f102
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This commit has been mentioned on Discourse Meta. There might be relevant details there:
https://meta.discourse.org/t/google-analytics-rss-pages-blocked/90861/2
f40f102
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This commit has been mentioned on Discourse Meta. There might be relevant details there:
https://meta.discourse.org/t/should-topic-rss-feeds-be-blocked-on-robots/137585/6