Implement the Go Wasm plugin: bot-detect

plugins/wasm-go/extensions/bot-detect/README.md

@@ -0,0 +1,58 @@
+<p>
+   <a href="README_EN.md"> English </a> | 中文
+</p>
+
+# 功能说明
+`bot-detect`插件可以用于识别并阻止互联网爬虫对站点资源的爬取
+
+# 配置字段
+
+| 名称 | 数据类型 | 填写要求 |  默认值 | 描述 |
+| -------- | -------- | -------- | -------- | -------- |
+|  allow     |  array of string     | 选填     |   -  |  配置匹配 User-Agent 请求头的正则表达式，匹配命中时将允许其访问   |
+|  deny     |  array of string     | 选填     |   -  |  配置匹配 User-Agent 请求头的正则表达式，匹配命中时将屏蔽请求   |
+|  blocked_code     |  number     | 选填     |   403  |  配置请求被屏蔽时返回的 HTTP 状态码   |
+|  blocked_message     |  string     | 选填     |   -  |  配置请求被屏蔽时返回的 HTTP 应答 Body   |
+
+`allow` 和 `deny` 字段可以均不配置，则执行默认的爬虫判断逻辑，通过配置 `allow` 字段可以将原本命中默认爬虫判断逻辑的请求放行，通过配置 `deny` 字段可以增加额外的爬虫判断逻辑。
+
+默认的爬虫判断正则表达式集合如下：
+
+```bash
+# Bots General matcher 'name/0.0'
+    (?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
+# Bots General matcher 'name 0.0'
+    (?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
+# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
+    ((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
+# Bots Pattern '/name-0.0'
+    /((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
+# Bots Pattern 'name/0.0'
+    \b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
+# More bots
+    (CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
+```
+
+# 配置示例
+
+## 放行原本命中爬虫规则的请求
+```yaml
+allow:
+- ".*Go-http-client.*"
+```
+
+若不作该配置，默认的 Golang 网络库请求会被视做爬虫，被禁止访问
+
+
+## 增加爬虫判断
+```yaml
+deny:
+- "spd-tools.*"
+```
+
+根据该配置，下列请求将被禁止访问：
+
+```bash
+curl http://example.com -H 'User-Agent: spd-tools/1.1'
+curl http://exmaple.com -H 'User-Agent: spd-tools'
+```

plugins/wasm-go/extensions/bot-detect/README_EN.md

@@ -0,0 +1,58 @@
+<p>
+   English | <a href="README.md">中文</a>
+</p>
+
+# Description
+`bot-detect` plugin can be used to identify and prevent web crawlers from crawling websites.
+
+# Configuration Fields
+
+| Name | Type | Requirement |  Default Value | Description |
+| -------- | -------- | -------- | -------- | -------- |
+|  allow     |  array of string     | Optional     |   -  |  A regular expression to match the User-Agent request header and will allow access if the match hits   |
+|  deny     |  array of string     | Optional     |   -  |  A regular expression to match the User-Agent request header and will block the request if the match hits   |
+|  blocked_code     |  number     | Optional     |   403  |  The HTTP status code returned when a request is blocked   |
+|  blocked_message     |  string     | Optional     |   -  |  The HTTP response Body returned when a request is blocked   |
+
+If field `allow` and field `deny` are not configured at the same time, the default logic to identify crawlers will be executed. By configuring the `allow` field, requests that would otherwise hit the default logic can be allowed. The judgement can be extended by configuring the `deny` field
+
+The default set of crawler judgment regular expressions is as follows：
+
+```bash
+# Bots General matcher 'name/0.0'
+    (?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
+# Bots General matcher 'name 0.0'
+    (?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
+# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
+    ((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
+# Bots Pattern '/name-0.0'
+    /((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
+# Bots Pattern 'name/0.0'
+    \b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
+# More bots
+    (CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
+```
+
+# Configuration Samples
+
+## Release Requests that would otherwise Hit the Crawler Rules
+```yaml
+allow:
+- ".*Go-http-client.*"
+```
+
+Without this configuration, the default Golang web library request will be treated as a crawler and access will be denied.
+
+
+## Add Crawler Judgement
+```yaml
+deny:
+- "spd-tools.*"
+```
+
+According to this configuration, the following requests will be denied:
+
+```bash
+curl http://example.com -H 'User-Agent: spd-tools/1.1'
+curl http://exmaple.com -H 'User-Agent: spd-tools'
+```

plugins/wasm-go/extensions/bot-detect/VERSION

@@ -0,0 +1 @@
+1.0.0

plugins/wasm-go/extensions/bot-detect/botdetect.yaml

@@ -0,0 +1,30 @@
+apiVersion: extensions.higress.io/v1alpha1
+kind: WasmPlugin
+metadata:
+  annotations:
+    higress.io/wasm-plugin-description: 用于识别并阻止互联网爬虫对站点资源的爬取
+    higress.io/wasm-plugin-title: Bot Detect
+  creationTimestamp: '2024-01-03T10:34:36Z'
+  generation: 2
+  labels:
+    higress.io/resource-definer: higress
+    higress.io/wasm-plugin-built-in: 'true'
+    higress.io/wasm-plugin-category: custom
+    higress.io/wasm-plugin-name: bot-detect
+    higress.io/wasm-plugin-version: 1.0.0
+  name: bot-detect
+  namespace: higress-system
+spec:
+  defaultConfigDisable: true
+  matchRules:
+    - config:
+        blocked_code: 401
+        blocked_message: a bot
+        deny:
+          - Chrome
+      configDisable: false
+      ingress:
+        - test
+  phase: AUTHN
+  priority: 310
+  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/20240103/bot-detect:1.0.0

plugins/wasm-go/extensions/bot-detect/config/bot_detect_config.go

@@ -0,0 +1,68 @@
+/*
+ * Copyright (c) 2022 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+	regexp "github.com/wasilibs/go-re2"
+)
+
+var DefaultBotRegex = []*regexp.Regexp{
+	regexp.MustCompile(`(\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(\.(\d+)(\.(\d+)|)|)`),
+	regexp.MustCompile(`((\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(\.(\d+)(\.(\d+)|)|))`),
+	regexp.MustCompile(`((([A-z0-9]{1,50}|[A-z\-]{1,} ?|)( the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(([ /]| v)(\d+)(\.(\d+)|)(\.(\d+)|)|))`),
+	regexp.MustCompile(`((Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(\.(\d+)(\.(\d+))?)?`),
+	regexp.MustCompile(`\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)`),
+	regexp.MustCompile(`((CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|))`),
+}
+
+type BotDetectConfig struct {
+	BlockedCode    uint32           `json:"blocked_code"`
+	BlockedMessage string           `json:"blocked_message"`
+	Allow          []*regexp.Regexp `json:"allow"`
+	Deny           []*regexp.Regexp `json:"deny"`
+}
+
+func (bdc *BotDetectConfig) FillDefaultValue() {
+	if bdc.BlockedCode == 0 {
+		bdc.BlockedCode = 403
+	}
+	if bdc.BlockedMessage == "" {
+		bdc.BlockedMessage = "Invalid User-Agent"
+	}
+}
+
+func (bdc *BotDetectConfig) Process(ua string) (bool, string) {
+	if ua == "" {
+		return false, "can not be empty"
+	}
+	for _, allowRule := range bdc.Allow {
+		if allowRule.MatchString(ua) {
+			return true, ""
+		}
+	}
+	for _, denyRule := range bdc.Deny {
+		if denyRule.MatchString(ua) {
+			return false, denyRule.String()
+		}
+	}
+	for _, defaultRule := range DefaultBotRegex {
+		if defaultRule.MatchString(ua) {
+			return false, defaultRule.String()
+		}
+	}
+	return true, ""
+}

plugins/wasm-go/extensions/bot-detect/config/bot_detect_config_test.go

@@ -0,0 +1,138 @@
+/*
+ * Copyright (c) 2022 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+	"github.com/stretchr/testify/assert"
+	regexp "github.com/wasilibs/go-re2"
+	"log"
+	"testing"
+)
+
+func toRegexMatch(regexs []string) []*regexp.Regexp {
+	re := make([]*regexp.Regexp, 0)
+	for _, regex := range regexs {
+		c, err := regexp.Compile(regex)
+		if err != nil {
+			log.Default().Fatal(err.Error())
+		}
+		re = append(re, c)
+	}
+	return re
+}
+
+func TestBotDetectConfig_ProcessTest(t *testing.T) {
+
+	tests := []struct {
+		name         string
+		ua           string
+		allow        []string
+		deny         []string
+		blockCode    uint32
+		blockMessage string
+		want         bool
+	}{
+		{
+			"test empty bot detect",
+			"",
+			[]string{},
+			[]string{},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test default bot detect",
+			"Ant-Tailsweep-1",
+			[]string{},
+			[]string{},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test default bot detect",
+			"indexer/1.2",
+			[]string{},
+			[]string{},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test default bot detect",
+			"indexer/1.1.0",
+			[]string{},
+			[]string{},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test default bot detect",
+			"YottaaMonitor",
+			[]string{},
+			[]string{},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test allow bot detect",
+			"BaiduMobaider",
+			[]string{"BaiduMobaider"},
+			[]string{},
+			401,
+			"bot has been blocked",
+			true,
+		},
+		{
+			"test deny bot detect",
+			"Chrome",
+			[]string{},
+			[]string{"Chrome"},
+			401,
+			"bot has been blocked",
+			false,
+		},
+		{
+			"test allow and deny bot detect",
+			"SameBotDetect",
+			[]string{"SameBotDetect"},
+			[]string{"SameBotDetect"},
+			401,
+			"bot has been blocked",
+			true,
+		},
+	}
+
+	for _, test := range tests {
+
+		t.Run(test.name, func(t *testing.T) {
+			bdc := BotDetectConfig{
+				BlockedCode:    test.blockCode,
+				BlockedMessage: test.blockMessage,
+				Allow:          toRegexMatch(test.allow),
+				Deny:           toRegexMatch(test.deny),
+			}
+			actual, _ := bdc.Process(test.ua)
+			assert.Equal(t, test.want, actual, "")
+		})
+
+	}
+
+}

plugins/wasm-go/extensions/bot-detect/go.mod

@@ -0,0 +1,23 @@
+module bot-detect
+
+go 1.19
+
+require (
+	github.com/alibaba/higress/plugins/wasm-go v1.3.2
+	github.com/stretchr/testify v1.8.0
+	github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
+	github.com/tidwall/gjson v1.14.3
+	github.com/wasilibs/go-re2 v1.4.1
+)
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
+	github.com/magefile/mage v1.15.0 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/tetratelabs/wazero v1.6.0 // indirect
+	github.com/tidwall/match v1.1.1 // indirect
+	github.com/tidwall/pretty v1.2.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)