Skip to content

Commit

Permalink
FEATURE: remove topic rss from robots
Browse files Browse the repository at this point in the history
Crawlers love hitting the rss feeds (confirmed that both Google and Bing do)

Experimenting with the impact of blocking these feeds and forcing Crawlers to hit
the content direct. It is better if they hit the actual page to start with as opposed to

1. Hit RSS feed
2. Find new content
3. Hit post link
4. Get canonical
5. Hit canonical

Lots of pointless work.

We do not know for sure what impact this will have on newsreader apps,
we will listen for feedback.
  • Loading branch information
SamSaffron committed Apr 11, 2018
1 parent abf0b1c commit f40f102
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions app/views/robots_txt/index.erb
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss

<% if @disallowed_user_agents %>
<% @disallowed_user_agents.each do |user_agent| %>
Expand Down

2 comments on commit f40f102

@discoursebot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/google-analytics-rss-pages-blocked/90861/2

@discoursebot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/should-topic-rss-feeds-be-blocked-on-robots/137585/6

Please sign in to comment.