Search engines index non-local communities, leading to undesirable results

[binwiederhier]

Thank you for your fantastic work on Lemmy. I love it!

Issue Summary

Due to the nature of the default robots.txt and the meta tags in Lemmy, search engines will index even non-local communities. This leads to results that are undesirable, such as unrelated/undesirable content being associated with your instance.

Example:

Steps to Reproduce

Open Google, and type <your instance> north korea

Suggested remediation/feature

I think it should be an opt-in feature to have non-local communities be indexed, e.g. [ ] Allow search engines to index non-local communities

Temporary workaround

I added this to my nginx config to prevent search engines from indexing the entire site:

# Disallow all search engines
location = /robots.txt {
    add_header Content-Type text/plain;
    return 200 "User-agent: *\nDisallow: /\n";
}

[jcgurango]

Correct me if I'm wrong but I believe this is the purpose of canonical URLs. Lemmy could add a link to the post in its original instance this way.

[kevinmershon]

@binwiederhier I think this may more appropriately be resolved within lemmy-ui: https://github.com/LemmyNet/lemmy-ui/pull/401/files

[binwiederhier]

@jcgurango I think that would certainly help, but honestly I don't want the pages to be associated with my instance at all. So a robots = noindex is preferable for these pages.

@kevinmershon Oh I see. There are two issue trackers. I'll close this one and re-open over there.

Reported in LemmyNet/lemmy-ui#1275