New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using a robots.txt file to prevent search engines from indexing old docs #7794

Open
astrofrog opened this Issue Sep 4, 2018 · 15 comments

Comments

Projects
None yet
9 participants
@astrofrog
Member

astrofrog commented Sep 4, 2018

According to rtfd/readthedocs.org#2430 we should be able to use robots.txt

@pllim

This comment has been minimized.

Show comment
Hide comment
@pllim

pllim Sep 5, 2018

Member

Why is this not a PR? 😉 😉 😉

Member

pllim commented Sep 5, 2018

Why is this not a PR? 😉 😉 😉

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Sep 5, 2018

Member

I don't actually know much about robots.txt files other than they exist 😉 I also am not 100% sure how this works for already published versions, i.e. do we need to go and add a robots.txt file to the branches for all already published versions/tags?

Member

astrofrog commented Sep 5, 2018

I don't actually know much about robots.txt files other than they exist 😉 I also am not 100% sure how this works for already published versions, i.e. do we need to go and add a robots.txt file to the branches for all already published versions/tags?

@pllim pllim added the sprint label Sep 5, 2018

@pllim

This comment has been minimized.

Show comment
Hide comment
@pllim

pllim Sep 5, 2018

Member

There used to be a "domain expert" label but we removed it, so I added a "sprint" label instead. Sounds like a good issue for Hacktoberfest that is coming up (https://hacktoberfest.digitalocean.com/).

Member

pllim commented Sep 5, 2018

There used to be a "domain expert" label but we removed it, so I added a "sprint" label instead. Sounds like a good issue for Hacktoberfest that is coming up (https://hacktoberfest.digitalocean.com/).

@bsipocz

This comment has been minimized.

Show comment
Hide comment
@bsipocz

bsipocz Sep 5, 2018

Member

as I recall we never actually created a domain expert label, but brainstormed about how to label these kind of issues. Adding hacktoberfest is actually quite a good idea, I was thinking about pitching astropy to non astronomy crowds.

Member

bsipocz commented Sep 5, 2018

as I recall we never actually created a domain expert label, but brainstormed about how to label these kind of issues. Adding hacktoberfest is actually quite a good idea, I was thinking about pitching astropy to non astronomy crowds.

@pllim pllim added Hacktoberfest and removed sprint labels Sep 6, 2018

@pllim

This comment has been minimized.

Show comment
Hide comment
@pllim

pllim Sep 6, 2018

Member

Okay, I created the label. 🤞

Member

pllim commented Sep 6, 2018

Okay, I created the label. 🤞

@Hackhard

This comment has been minimized.

Show comment
Hide comment
@Hackhard

Hackhard Oct 9, 2018

I would love to take on this issue as part of hacktoberfest 2018. Where should I be starting with? Also through the comments mentioned above, do correct me if I am wrong, I need to address the file at http://docs.astropy.org/robots.txt to work on this issue, right?

Hackhard commented Oct 9, 2018

I would love to take on this issue as part of hacktoberfest 2018. Where should I be starting with? Also through the comments mentioned above, do correct me if I am wrong, I need to address the file at http://docs.astropy.org/robots.txt to work on this issue, right?

@dasdachs

This comment has been minimized.

Show comment
Hide comment
@dasdachs

dasdachs Oct 10, 2018

Contributor

Here's my pull request for this issue.

Contributor

dasdachs commented Oct 10, 2018

Here's my pull request for this issue.

@Celebrimbor-forge

This comment has been minimized.

Show comment
Hide comment
@Celebrimbor-forge

Celebrimbor-forge Oct 16, 2018

Add another 'assignment' readme example

Celebrimbor-forge commented Oct 16, 2018

Add another 'assignment' readme example

@pllim

This comment has been minimized.

Show comment
Hide comment
@pllim

pllim Oct 16, 2018

Member

Seems like this is already resolved by #7874 ? Thanks, @dasdachs !

Member

pllim commented Oct 16, 2018

Seems like this is already resolved by #7874 ? Thanks, @dasdachs !

@good-praxis

This comment has been minimized.

Show comment
Hide comment
@good-praxis

good-praxis Oct 17, 2018

If it has been solved, why not close the issue? 😜

good-praxis commented Oct 17, 2018

If it has been solved, why not close the issue? 😜

@bsipocz

This comment has been minimized.

Show comment
Hide comment
@bsipocz

bsipocz Oct 17, 2018

Member

well, the issue has been resolved partially only, thus we keep it open.

Member

bsipocz commented Oct 17, 2018

well, the issue has been resolved partially only, thus we keep it open.

@breofca

This comment has been minimized.

Show comment
Hide comment
@breofca

breofca Oct 19, 2018

@bsipocz could you elaborate?

breofca commented Oct 19, 2018

@bsipocz could you elaborate?

@pllim

This comment has been minimized.

Show comment
Hide comment
@pllim

pllim Oct 19, 2018

Member

has been resolved partially

To elaborate, we are waiting for upstream fix over at RTD. I'll update the labels. Sorry for the confusion and thank you for your interest (feel free to look at other "good first issues" if you wish to contribute).

Member

pllim commented Oct 19, 2018

has been resolved partially

To elaborate, we are waiting for upstream fix over at RTD. I'll update the labels. Sorry for the confusion and thank you for your interest (feel free to look at other "good first issues" if you wish to contribute).

@astrofrog

This comment has been minimized.

Show comment
Hide comment
@astrofrog

astrofrog Oct 19, 2018

Member

@breofca @good-praxis - see my comment in rtfd/readthedocs.org#3161 (comment) - while #7874 might work going forward, it doesn't solve the problem that all the existing tagged versions (e.g. v1.0) don't have these meta tags and we'd need to re-tag all the versions (which we won't do).

Member

astrofrog commented Oct 19, 2018

@breofca @good-praxis - see my comment in rtfd/readthedocs.org#3161 (comment) - while #7874 might work going forward, it doesn't solve the problem that all the existing tagged versions (e.g. v1.0) don't have these meta tags and we'd need to re-tag all the versions (which we won't do).

@ambelis2

This comment has been minimized.

Show comment
Hide comment
@ambelis2

ambelis2 Oct 22, 2018

Editing in Sphinx with headers with sub pages with labeling should maximize robots.txt to minimize number of outputs of old doc text.

ambelis2 commented Oct 22, 2018

Editing in Sphinx with headers with sub pages with labeling should maximize robots.txt to minimize number of outputs of old doc text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment