Skip to content

A list of public sites that are blocking ChatGPT from crawling

Notifications You must be signed in to change notification settings

zuccs/disallow-chatgpt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 

Repository files navigation

Sites Officially Disallowing ChatGPT bot

Now that OpenAI has officially recognised GPTbot, here is a list of public sites that have taken the iniatitive to block ChatGPT from scraping their site going forward.

Site Source Date Found DR
The Verge https://www.theverge.com/robots.txt 12th August 2023 92
Vox https://www.vox.com/robots.txt 12th August 2023 91
Stack Overflow https://stackoverflow.com/robots.txt 14th August 2023 92
Ikea https://www.ikea.com/robots.txt 14th August 2023 91
Polygon https://www.polygon.com/robots.txt 14th August 2023 86
Quora https://www.quora.com/robots.txt 14th August 2023 91
Airbnb https://www.airbnb.com/robots.txt 14th August 2023 92
TPT https://www.teacherspayteachers.com/robots.txt 14th August 2023 86
BAILII https://www.bailii.org/robots.txt 15th August 2023 82
W Magazine https://www.wmagazine.com/robots.txt 15th August 2023 81
MyHeritage https://www.myheritage.com/robots.txt 15th August 2023 79
POPSUGAR https://www.popsugar.com/robots.txt 15th August 2023 88
Economic Times https://economictimes.indiatimes.com/robots.txt 15th August 2023 91
Mastodon https://github.com/mastodon/mastodon/blob/main/public/robots.txt 16th August 2023 N/A
Android Authority https://www.androidauthority.com/robots.txt 17th August 2023 85
Glassdoor https://www.glassdoor.com/robots.txt 17th August 2023 91
Basketball Reference https://www.basketball-reference.com/robots.txt 17th August 2023 78
Sports Reference https://www.sports-reference.com/robots.txt 17th August 2023 76
Coursera https://www.coursera.org/robots.txt 17th August 2023 91
SB Nation https://www.sbnation.com/robots.txt 17th August 2023 83
Outsports https://www.outsports.com/robots.txt 17th August 2023 74
CarsDirect https://www.carsdirect.com/robots.txt 17th August 2023 72
SteamDB https://steamdb.info/robots.txt 17th August 2023 76
Amazon https://www.amazon.com/robots.txt 18th August 2023 96
Reuters https://www.reuters.com/robots.txt 18th August 2023 92
Radio France https://www.radiofrance.fr/robots.txt 18th August 2023 89
Video Gamer https://www.videogamer.com/robots.txt 18th August 2023 71
The Athletic https://theathletic.com/robots.txt 18th August 2023 83
The New York Times https://www.nytimes.com/robots.txt 19th August 2023 94
Indeed https://www.indeed.com/robots.txt 19th August 2023 92
Bustle https://www.bustle.com/robots.txt 19th August 2023 89
Curbed https://www.curbed.com/robots.txt 19th August 2023 87
BBC Good Food https://www.bbcgoodfood.com/robots.txt 21st August 2023 87
DigitalOceean https://www.digitalocean.com/robots.txt 21st August 2023 91
AIAA https://www.aiaa.org/robots.txt 21st August 2023 79
Kelley Blue Book https://www.kbb.com/robots.txt 23rd August 2023 89
Autotrader https://www.autotrader.com/robots.txt 23rd August 2023 80
The Ringer https://www.theringer.com/robots.txt 23rd August 2023 81
Healthgrades https://www.healthgrades.com/robots.txt 25th August 2023 87
Chicago Tribune https://www.chicagotribune.com/robots.txt 25th August 2023 91
Bloomberg https://www.bloomberg.com/robots.txt 25th August 2023 92
Insider https://www.insider.com/robots.txt 25th August 2023 90
ESPN https://www.espn.com/robots.txt 28th August 2023 91
TED https://www.ted.com/robots.txt 28th August 2023 92
Cisco https://www.cisco.com/robots.txt 28th August 2023 91
WorldCat https://www.worldcat.org/robots.txt 28th August 2023 91
Simply Psychology https://www.simplypsychology.org/robots.txt 28th August 2023 83
Eater https://www.eater.com/robots.txt 28th August 2023 90
Business Insider https://www.businessinsider.com/robots.txt 28th August 2023 92
Vultr https://www.vultr.com/robots.txt 28th August 2023 86
Nikkei https://www.nikkei.com/robots.txt 28th August 2023 91
Vogue https://www.vogue.com/robots.txt 30th August 2023 90
Bon Appétit https://www.bonappetit.com/robots.txt 30th August 2023 90
Bankrate https://www.bankrate.com/robots.txt 30th August 2023 90
ABC News https://abcnews.go.com/robots.txt 30th August 2023 92
ABC https://www.abc.net.au/robots.txt 30th August 2023 91

About

A list of public sites that are blocking ChatGPT from crawling

Resources

Stars

Watchers

Forks