Skip to content

Add Mistral and DeepSeek bot detection to ai-bot-monitor#113

Merged
jasperf merged 1 commit into
mainfrom
add/mistral-deepseek-bot-detection
Jun 26, 2026
Merged

Add Mistral and DeepSeek bot detection to ai-bot-monitor#113
jasperf merged 1 commit into
mainfrom
add/mistral-deepseek-bot-detection

Conversation

@jasperf

@jasperf jasperf commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Expands scripts/monitoring/ai-bot-monitor.sh to detect three previously untracked AI crawlers:

  • MistralAI-User — Mistral's live Le Chat fetcher
  • MistralAI-Index — Mistral's indexing crawler
  • DeepSeekBot — DeepSeek's web crawler

All three are added to both the AI_BOTS display array and the aggregate AI_PATTERN matcher, so they now show up in per-bot request/bandwidth counts, scraped-page lists, and robots.txt compliance checks.

Why

Verified against ~2 months of production access logs across two WordPress sites:

  • DeepSeekBot was actively crawling — real page fetches plus probes for /.env.production and framework build manifests — but went completely uncounted because the script never matched it.
  • Neither Mistral bot has appeared at all (Mistral appears to rely on Common Crawl rather than running its own broad crawl), so reports were silently blind to whether it ever does.

Changes

  • scripts/monitoring/ai-bot-monitor.sh — add the three UA patterns to AI_BOTS + AI_PATTERN
  • CHANGELOG.md[2.15.0] - 2026-06-26

Test

  • bash -n scripts/monitoring/ai-bot-monitor.sh passes (no syntax errors)

@jasperf jasperf merged commit b82a02d into main Jun 26, 2026
@jasperf jasperf deleted the add/mistral-deepseek-bot-detection branch June 26, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant