Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

block all robots in robots.txt #1

Merged
merged 1 commit into from Apr 8, 2017

Conversation

Projects
None yet
3 participants
@nolanlawson
Copy link
Member

commented Apr 8, 2017

We can change this later, e.g. to configure it on a per-user basis, but defaulting to most-private seems best.

@nolanlawson

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2017

Merging, just opening the PR for transparency

@nolanlawson nolanlawson merged commit c289bb3 into tootcafe Apr 8, 2017

@TekBear

This comment has been minimized.

Copy link

commented Apr 13, 2017

I think you should not block archive.org crawler - recording history is important

nolanlawson pushed a commit that referenced this pull request Apr 14, 2017

nolanlawson added a commit that referenced this pull request Apr 14, 2017

nolanlawson added a commit that referenced this pull request Apr 17, 2017

@nolanlawson

This comment has been minimized.

Copy link
Member Author

commented Apr 28, 2017

@TekBear I'd need to ask what the community thinks. If they're okay with it, then I'm okay with it. :)

nolanlawson added a commit that referenced this pull request Apr 29, 2017

nolanlawson added a commit that referenced this pull request May 23, 2017

nolanlawson added a commit that referenced this pull request May 28, 2017

nolanlawson added a commit that referenced this pull request Jun 15, 2017

nolanlawson added a commit that referenced this pull request Jul 28, 2017

nolanlawson added a commit that referenced this pull request Jul 31, 2017

nolanlawson added a commit that referenced this pull request Aug 1, 2017

nolanlawson added a commit that referenced this pull request Aug 7, 2017

nolanlawson added a commit that referenced this pull request Sep 10, 2017

nolanlawson added a commit that referenced this pull request Sep 17, 2017

nolanlawson added a commit that referenced this pull request Oct 13, 2017

nolanlawson added a commit that referenced this pull request Oct 18, 2017

nolanlawson added a commit that referenced this pull request Oct 19, 2017

nolanlawson added a commit that referenced this pull request Dec 16, 2017

nolanlawson added a commit that referenced this pull request Jan 6, 2018

nolanlawson added a commit that referenced this pull request Jan 10, 2018

@nemobis

This comment has been minimized.

Copy link

commented Aug 31, 2018

Did you mean to block the Internet Archive Wayback machine as well?

nolanlawson added a commit that referenced this pull request Sep 16, 2018

nolanlawson added a commit that referenced this pull request Oct 13, 2018

@nolanlawson

This comment has been minimized.

Copy link
Member Author

commented Oct 29, 2018

Yes. This is a block of everything.

nolanlawson added a commit that referenced this pull request Nov 10, 2018

nolanlawson added a commit that referenced this pull request Dec 31, 2018

nolanlawson added a commit that referenced this pull request Jan 20, 2019

nolanlawson added a commit that referenced this pull request Feb 6, 2019

nolanlawson added a commit that referenced this pull request Feb 24, 2019

nolanlawson added a commit that referenced this pull request May 4, 2019

nolanlawson added a commit that referenced this pull request May 5, 2019

nolanlawson added a commit that referenced this pull request May 28, 2019

nolanlawson added a commit that referenced this pull request Jun 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.