Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
the page at docs/search_results.html should be more hidden (exclude it from search results; add to a robots.txt) #1027
Have you tried searching for something when you arrive at that page?
Hmm, thanks @ebbeck - you found a bug in the structure of our site, I think.
It looks like the file at docs/search_results.html is not meant for people to find. Instead it is a template used in the search box at the top of all pages. I'll update the title of this issue to track the underlying problem.
Thanks @Aaranged - eagle eyed as ever. In this case @RichardWallis and I decided it was best not to confuse things by having the webschemas draft site show up. Depending on whether the site is running in "official" mode or webschemas-etc mode, we serve a different robots.txt - https://github.com/schemaorg/schemaorg/blob/sdo-callisto/docs/robots-blockall.txt vs https://github.com/schemaorg/schemaorg/blob/sdo-callisto/docs/robots.txt
The goryAppEngine details are in the corresponding *.yaml files. Amongst other things, the official site version should serve a simple sitemap...
the recommended method to block indexing a page is meta tags.
We should add :
Diff between the two methods :