Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to block google from indexing yourl? #2202

Closed
Ruriko opened this Issue Feb 20, 2017 · 5 comments

Comments

Projects
None yet
4 participants
@Ruriko
Copy link

Ruriko commented Feb 20, 2017

How do I make search engines not to index any yourl links?

@TuKod

This comment has been minimized.

Copy link

TuKod commented Mar 1, 2017

Why would you want to?

By allowing Search Engines to crawl your YOURLS site, will transfer link juice from the short url to the long one.

To prevent honest Search Engines from crawling your whole site, or even one URL, you could use robots.txt (Google robots.txt for info).

Regards,

Lan

@ozh

This comment has been minimized.

Copy link
Member

ozh commented Mar 2, 2017

YOURLS links are not "indexed". If Googlebot finds a YOURLS link, eg a page somewhere or a tweet contains http://sho.rt/blah, Googlebot will follow it and, if allowed by the destination, index and archive the destination page.

YOURLS links are only redirects, they are not indexed by Google.

If your question is "how do I prevent Google from following YOURLS links", answer is "make a custom plugin that checks user agent before redirection to prevent googlebot from being redirected"

@ozh ozh closed this Mar 2, 2017

@Sushubh

This comment has been minimized.

Copy link

Sushubh commented Jul 27, 2018

Just wanted to mention here that Google is indexing yourls short links for me.

https://www.google.co.in/search?q=site%3At.broadbandforum.co

I added robots.txt file few days ago but it only seems to prevent indexing of content (on some links but not all at the moment) but the links still make it to Google search results.

Google suggest adding to meta tags which is not applicable in case of redirected links. The other option is the X-Robots-Tag HTTP header... implementation of which is outside my area of expertise. Just wanted to post this here in case some developer would consider making a plugin that can do this?

This is not that big of a deal for me except that it seems to pollute the site search results for my domain as I am using the same domain (subdomain) for YOURLS. I am not sure if it is hurting my site's ranking/authority in Google's eyes as these links end up redirecting to other domains :) Cheers.

@ozh

This comment has been minimized.

Copy link
Member

ozh commented Aug 4, 2018

Indeed. I think Googlebot has changed behavior over past years.

Quick research makes me think too we may add PHP header header("X-Robots-Tag: noindex", true); to every redirection page but I'm not sure it's not outdated.

@Sushubh

This comment has been minimized.

Copy link

Sushubh commented Aug 5, 2018

https://developers.google.com/search/reference/robots_meta_tag does not indicate that these are deprecated.

ozh added a commit that referenced this issue Aug 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.