Doesn't seem to be indexed well by search engines

[tomjaguarpaw]

@jeukshi reports that Googling for “GHC-39999” (with and without quotes) doesn't return any results from the error index despite it being a valid page: https://errors.haskell.org/messages/GHC-39999/
Do we have any idea why the site might not be indexed properly?

[Bodigrim]

Even more worryingly, https://www.google.com/search?q=GHC-39999+site%3Aerrors.haskell.org is empty.

[BinderDavid]

The issue seems to be the following line in the template:

error-message-index/message-index/templates/default.html

Line 8 in 6b513c8

<meta name="robots" content="noindex">

which was added explicitly in 36d8c95 by @serras

Do you recall why you added this line?

[tomjaguarpaw]

The PR was here: haskell/error-messages#92

Looks like it was made alongside a large number of PRs during ZuriHac 2022: https://zfoh.ch/zurihac2022/

Here's the documentation of noindex: https://en.wikipedia.org/wiki/Noindex

Reasons why one might want to use this meta tag include ... web pages that are under development

Perhaps the intention was to temporarily disable indexing and re-enable it after ZuriHac?

[tomjaguarpaw]

So far only three pages are indexed in the domain: https://www.google.com/search?q=+site%3Aerrors.haskell.org

Perhaps it takes some time?

[Bodigrim]

Google provides a few instruments to monitor, like https://search.google.com/search-console/welcome

Can Hakyll generate Sitemap file? https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap

[tomjaguarpaw]

Five pages are indexed now, so something is happening...

https://www.google.com/search?q=+site%3Aerrors.haskell.org

[tomjaguarpaw]

Now nine, so something is happening, but very slowly.

[BinderDavid]

I have no experience with SEO whatsoever, maybe we could boost this issue in some channel with people who have some experience in this. Maybe one short look is all it takes?

[tomjaguarpaw]

Same here. Maybe we can ask for help on Discourse and go from there?

[Bodigrim]

I suggested some pointers above in #537 (comment), although my SEO experience is 15 years old :)

[BinderDavid]

I think we are really missing a sitemap. Thanks @Bodigrim !
Google's documentation mentions several properties under which it makes sense to implement one: https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview These all seem to apply to our case.

I also found this tutorial for Hakyll https://www.rohanjain.in/hakyll-sitemap/ and will implement that.