Add toggle for blocking search indexing of Grav URL parameters

[nbusseneau]

When enabled, a noindex <meta> tag will be added to pages with Grav URL parameters (e.g. /blog/foo:bar).

Rationale: avoid page duplicates being indexed by search engines.

I noticed this was happening when parameterized URLs were scrapable by search engines, which is the case by default on Quark with sidebar using Taxonomy List and Archives plugins.

It seems that at least Google treats them as separate pages, even when properly using canonical link elements and sitemaps.

As specified in Google's documentation, the indexer may choose to ignore content-provided indicators:

Google chooses the canonical page based on a number of factors (or signals), such as whether the page is served via http or https; page quality; presence of the URL in a sitemap; and any "rel=canonical" labeling. You can indicate your preference to Google using these techniques, but Google may choose a different page as canonical than you do, for various reasons.

This behavior might be undesirable. In this case, the only resort is to block indexing altogether using a noindex <meta> tag:

<meta name="robots" content="noindex">

Obviously, we'd only want to block indexing for Grav parameterized URLs. Including the <meta> tag in a base template is not suitable: it must be conditionally inserted.

Hence this toggle.

[nbusseneau]

@rhukster Revisiting old PRs -- what do you think about this? I have been running a custom Quark theme on my end with this switch for a while, and find it really useful to not have Google index Grav parameterized URLs on my website.

[rhukster]

Rather than this being a toggle in every theme, it makes more sense to me if this logic was rewritten as a plugin. I think that would be more useful and not theme dependent. What do you think?

[nbusseneau]

Oooh that is a nice idea, I did not think about this. I agree!

[NicoHood]

Isnt the canonial url for such usecases?

[nbusseneau]

@NicoHood As said in the OP:

It seems that at least Google treats them as separate pages, even when properly using canonical link elements and sitemaps.

As specified in Google's documentation, the indexer may choose to ignore content-provided indicators:

Google chooses the canonical page based on a number of factors (or signals), such as whether the page is served via http or https; page quality; presence of the URL in a sitemap; and any "rel=canonical" labeling. You can indicate your preference to Google using these techniques, but Google may choose a different page as canonical than you do, for various reasons.

I have verified this happening with Google indexing my website's parameterized URLs as separate pages even with the proper canonical URL specified.