Mirror ShellCheck wiki somewhere search engines can index

[dimo414]

I'm sure this is a known issue, but I searched and didn't see anything relevant in the ShellCheck repo; sorry if this is noisy

Unfortunately, due to community/community#4992 the ShellCheck wiki doesn't show up in Google for searches like [SC2086]; https://github-wiki-see.page/ sometimes shows up in its place, but it's a poor substitute (for the longest time I thought it was a spam mirror like you see for StackOverflow questions and intentionally avoided clicking on it).

It's possible GitHub will eventually lift this restriction or change the requirements to allow the ShellCheck wiki to be indexable once more, but I'd argue it'd still be better to host this important documentation elsewhere (e.g. on ShellCheck.net) even if GitHub's policy changes.

[nelsonjchen]

I run https://github-wiki-see.page/ so I'm very aware of the current situation.

Here's another option as well. I'm just presenting it but I don't have a preference for it or any other solution.

Since this repository has so many stars (more than 1000+, by a lot!), an approach like what the VSCode for Go extension people did is possible as well if one wants to "preserve" the GitHub Wiki UI/URLs:

golang/vscode-go#2094

Wikis on 1000 star+ repositories can be indexed if "Editing Restricted to Repository Collaborators" is turned on. What the VSCode for Go people did was take advantage of that to deploy to the Wiki from their own GitHub repository. The drawback is that users need to submit pull requests.

[nelsonjchen]

A possible way to lessen the hit of locking down the Wiki might be to synthesize Edit on GitHub links into the markdown on the Wiki like what I've suggested in that issue.

[koalaman]

ShellCheck already had redirections from e.g. https://www.shellcheck.net/wiki/SC2086 to the GitHub wiki. I've replaced the redirections with statically generated pages. Hopefully the formatting isn't too broken, and they'll be indexed by search engines soon.

[nelsonjchen]

@koalaman

Those pages may be a bit invisible to search engines at the moment as they aren't linked anywhere, accessible some sort of index page of some sort, or visible through a sitemap. Users may be visiting them through browsers after copy/clicking them from the terminal but otherwise they aren't very "visible" to crawlers.

Adding a sitemap would probably be the quickest way to get them into the indexes of many search engines. Being able to get to them from the front page by clicking through web pages would also probably help their ranking a lot as well.

You can measure how indexed those new static pages are at least on Google by looking at the https://search.google.com/search-console and adding a sitemap.

Here's a glimpse into GHWSEE's debug view of its effectiveness on shellcheck's GitHub Wiki stuff as seen through Google's Search Console:

Equivalent systems exist for Bing (which also provide much of their index to DuckDuckGo).

---

By the way, I cannot figure out a way to automatically disable GHWSEE for GitHub repos/Wikis with this kind of arrangement as there's no way to detect that a project has decided to mirror their Wiki onto an indexable site non-expensively. Any way that looks more official and kicks the service I built's ranking down will work for shellcheck's needs while I don't have this manually disabled in GHWSEE.

I'm looking forward to adding shellcheck to the blacklist. I want to wait until you have that indexing issue solved and monitored though.

[koalaman]

Don't worry, there's a complete index on https://www.shellcheck.net/wiki/ plus https://www.shellcheck.net/wiki/sitemap.txt which I already submitted to Google

[nelsonjchen]

Solved and monitored! Cool! I'll blacklist when I am able to then. 👍