Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mirror ShellCheck wiki somewhere search engines can index #2487

Closed
dimo414 opened this issue Apr 14, 2022 · 6 comments
Closed

Mirror ShellCheck wiki somewhere search engines can index #2487

dimo414 opened this issue Apr 14, 2022 · 6 comments

Comments

@dimo414
Copy link
Contributor

dimo414 commented Apr 14, 2022

I'm sure this is a known issue, but I searched and didn't see anything relevant in the ShellCheck repo; sorry if this is noisy

Unfortunately, due to community/community#4992 the ShellCheck wiki doesn't show up in Google for searches like [SC2086]; https://github-wiki-see.page/ sometimes shows up in its place, but it's a poor substitute (for the longest time I thought it was a spam mirror like you see for StackOverflow questions and intentionally avoided clicking on it).

It's possible GitHub will eventually lift this restriction or change the requirements to allow the ShellCheck wiki to be indexable once more, but I'd argue it'd still be better to host this important documentation elsewhere (e.g. on ShellCheck.net) even if GitHub's policy changes.

@nelsonjchen
Copy link

I run https://github-wiki-see.page/ so I'm very aware of the current situation.

Here's another option as well. I'm just presenting it but I don't have a preference for it or any other solution.

Since this repository has so many stars (more than 1000+, by a lot!), an approach like what the VSCode for Go extension people did is possible as well if one wants to "preserve" the GitHub Wiki UI/URLs:

golang/vscode-go#2094

Wikis on 1000 star+ repositories can be indexed if "Editing Restricted to Repository Collaborators" is turned on. What the VSCode for Go people did was take advantage of that to deploy to the Wiki from their own GitHub repository. The drawback is that users need to submit pull requests.

@nelsonjchen
Copy link

A possible way to lessen the hit of locking down the Wiki might be to synthesize Edit on GitHub links into the markdown on the Wiki like what I've suggested in that issue.

koalaman added a commit to koalaman/shellcheck.net that referenced this issue May 11, 2022
@koalaman
Copy link
Owner

ShellCheck already had redirections from e.g. https://www.shellcheck.net/wiki/SC2086 to the GitHub wiki. I've replaced the redirections with statically generated pages. Hopefully the formatting isn't too broken, and they'll be indexed by search engines soon.

@nelsonjchen
Copy link

nelsonjchen commented May 11, 2022

@koalaman

Those pages may be a bit invisible to search engines at the moment as they aren't linked anywhere, accessible some sort of index page of some sort, or visible through a sitemap. Users may be visiting them through browsers after copy/clicking them from the terminal but otherwise they aren't very "visible" to crawlers.

Adding a sitemap would probably be the quickest way to get them into the indexes of many search engines. Being able to get to them from the front page by clicking through web pages would also probably help their ranking a lot as well.

You can measure how indexed those new static pages are at least on Google by looking at the https://search.google.com/search-console and adding a sitemap.

Here's a glimpse into GHWSEE's debug view of its effectiveness on shellcheck's GitHub Wiki stuff as seen through Google's Search Console:

image

Equivalent systems exist for Bing (which also provide much of their index to DuckDuckGo).


By the way, I cannot figure out a way to automatically disable GHWSEE for GitHub repos/Wikis with this kind of arrangement as there's no way to detect that a project has decided to mirror their Wiki onto an indexable site non-expensively. Any way that looks more official and kicks the service I built's ranking down will work for shellcheck's needs while I don't have this manually disabled in GHWSEE.

I'm looking forward to adding shellcheck to the blacklist. I want to wait until you have that indexing issue solved and monitored though.

@koalaman
Copy link
Owner

Don't worry, there's a complete index on https://www.shellcheck.net/wiki/ plus https://www.shellcheck.net/wiki/sitemap.txt which I already submitted to Google

@nelsonjchen
Copy link

Solved and monitored! Cool! I'll blacklist when I am able to then. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants