Releases: ai-robots-txt/ai.robots.txt
Releases · ai-robots-txt/ai.robots.txt
Release list
v1.50: Add some bots and improve pattern matching
What's Changed
- Pass commit messages to bash via env, not template interpolation by @INXPRNCD in #254
- Fill "Unclear at this time." from primary operator docs for 6 entries by @INXPRNCD in #255
- robots.py: duplicate parser = argparse.ArgumentParser(...) by @guest20 in #256
- do not needlessly escape hypens, make nginx matches case-sensitive, further minify regexps by @girst in #259
- fix lighttpd config for 1.x by @girst in #261
- fix(code): unanchor PCRE patterns to allow matching real user-agent headers (#257) by @Ilyan321 in #260
- Add Bot Ledger to Related resources by @farrelldan in #262
- add Lightpanda user agent by @acook in #263
New Contributors
- @girst made their first contribution in #259
- @Ilyan321 made their first contribution in #260
- @farrelldan made their first contribution in #262
- @acook made their first contribution in #263
Full Changelog: v1.49...v1.50
Contributors
acook, guest20, and 4 other contributors
Assets 3
0
Join discussion
v1.49: delete IbouBot
Contributors
glyn
Assets 3
v1.48: improve matching in auxiliary config files
What's Changed
- Add: Mozilla-Tabstack by @bobmatyas in #240
- Anchor generic Spider user-agent match by @iamdhrv in #241
New Contributors
- @bobmatyas made their first contribution in #240
- @iamdhrv made their first contribution in #241
Full Changelog: v1.47...v1.48
Contributors
iamdhrv and bobmatyas
Assets 3
v1.47: Rename dark visitors to known agents and add AgentTimes
What's Changed
- add: AgentTimes bot by @justinribeiro in #232
- robots.py: fix darkvisiters descriptions and links by @guest20 in #236
- Renaming Dark Visitors to Known Agents by @ghking in #238
New Contributors
- @justinribeiro made their first contribution in #232
- @guest20 made their first contribution in #236
- @ghking made their first contribution in #238
Full Changelog: v1.46...v1.47
Contributors
justinribeiro, ghking, and guest20
Assets 3
Add more dark visitor bot categories
What's Changed
- Add NagetBot and newsai user-agents by @gsauthof in #227
- Add missing bot categories by @axeleroy in #229
- Keep generated robot files synced after Dark Visitors update by @cfcryptotrader-rgb in #231
New Contributors
- @gsauthof made their first contribution in #227
- @axeleroy made their first contribution in #229
- @cfcryptotrader-rgb made their first contribution in #231
Full Changelog: v1.45...v1.46
Contributors
gsauthof, axeleroy, and cfcryptotrader-rgb
Assets 3
v1.45: Add Traefik info, LAIONDownloader, and lighttpd config
What's Changed
- feat: allow robots access to
/robots.txtin nginx by @fiskhandlarn in #195 - Create traefik-manual-setup.md by @maxheadroom in #130
- Add requirements.txt with project dependencies by @Anshita-18H in #201
- Tidy README by @glyn in #202
- Standardise editor options by @glyn in #199
- Clarify that DeepSeekBot does not respect robots.txt by @glyn in #200
- Adding LAIONDownloader by @newbold in #206
- Fix/editorconfig by @fiskhandlarn in #205
- Lighttpd by @flymarq in #216
New Contributors
- @fiskhandlarn made their first contribution in #195
- @maxheadroom made their first contribution in #130
- @Anshita-18H made their first contribution in #201
- @newbold made their first contribution in #206
- @flymarq made their first contribution in #216
Full Changelog: v1.44...v1.45
Contributors
glyn, maxheadroom, and 4 other contributors
Assets 3
v1.44: corrects amazon-kendra, adds release documentation
What's Changed
Full Changelog: v1.43...v1.44
Contributors
glyn
Assets 3
v1.43: adds IbouBot, Linguee Bot, BuddyBot, AmazonBuyForMe, atlassian-bot and others
What's Changed
- chore: adds IbouBot by @cdransf in #185
- Use third party in text, rather than first by @glyn in #186
- chore: adds Linguee Bot by @cdransf in #187
- chore: adds BuddyBot by @cdransf in #188
- chore(robots.json): add AmazonBuyForMe and atlassian-bot by @cdransf in #189
- chore: adds NotebookLM and KlaviyoAIBot agents by @cdransf in #192
Full Changelog: v1.42...v1.43
Contributors
glyn and cdransf
Assets 3
v1.42: add Google-NotebookLM , amazon-kendra-, Anomura, Cloudflare-AutoRAG and Bravebot
What's Changed
Full Changelog: v1.41...v1.42
Contributors
cdransf
