Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow robots to crawl and index manuals #138

Closed
wants to merge 1 commit into from
Closed

Allow robots to crawl and index manuals #138

wants to merge 1 commit into from

Conversation

@fofr
Copy link
Contributor

@fofr fofr commented Mar 19, 2015

Revert bcf09b9 (#36) so that Google and other bots can index and crawl manuals.

cc @evilstreak

Revert bcf09b9
(#36), so Google and
other bots can index and crawl manuals.
@fofr
Copy link
Contributor Author

@fofr fofr commented Mar 19, 2015

Temporarily closing while discussing whether bots should be allowed to index all manuals.

@fofr fofr closed this Mar 19, 2015
@rooreynolds
Copy link

@rooreynolds rooreynolds commented Apr 14, 2015

We are the canonical source of information for published manuals, and it makes sense to let the spiders in for
https://www.gov.uk/guidance/convert-to-an-academy-information-for-schools
https://www.gov.uk/guidance/digital-and-technology-skills
https://www.gov.uk/guidance/content-design
https://www.gov.uk/guidance/contact-the-government-digital-service
https://www.gov.uk/guidance/style-guide
https://www.gov.uk/guidance/how-to-publish-on-gov-uk
etc

BUT, we're not the canonical source for the HMRC manuals yet. They're intentionally beta at the moment, and we definitely wouldn't want people stumbling across them.

I think we should set things up to allow:

  • non-HMRC manuals to be crawled (ie everything at /guidance = allowed)
  • and HMRC manuals to remain disallowed for now, until they agree to redirect to them and they become the canonical source (ie everything under /hmrc-internal-manuals = a spider free zone for now)
@rboulton
Copy link
Contributor

@rboulton rboulton commented Apr 14, 2015

That sounds ideal, @rooreynolds

@rooreynolds
Copy link

@rooreynolds rooreynolds commented Apr 15, 2015

Thanks. I notice this pull request is currently closed though. I've created a new story in the Core backlog to do this work soon. (https://www.pivotaltracker.com/story/show/92516170) Is someone else is already case though?

@barrucadu barrucadu deleted the allow-robots branch Sep 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.