Skip to content

feat: add toolset plugin#13444

Open
AlinsRan wants to merge 7 commits into
masterfrom
feat/toolset-plugin
Open

feat: add toolset plugin#13444
AlinsRan wants to merge 7 commits into
masterfrom
feat/toolset-plugin

Conversation

@AlinsRan
Copy link
Copy Markdown
Contributor

@AlinsRan AlinsRan commented May 26, 2026

Summary

Add the toolset plugin — a diagnostics and observability framework that hosts multiple lightweight sub-plugins. Each sub-plugin is configured via plugin_attr.toolset in config.yaml and is dynamically loaded/unloaded at runtime without restarting APISIX.

Sub-plugins

trace

Instruments APISIX request phases and emits a formatted timing table to the error log for sampled requests.

Features:

  • Configurable sampling rate (N out of 100 requests)
  • Host and path allowlist filtering (glob patterns)
  • Trace header detection (x-request-id, traceparent, sw8, x-b3-traceid)
  • Optional UUID generation when no trace header is present
  • Minimum timespan threshold to filter out fast requests
  • Additional nginx/APISIX variable capture

table_count

Periodically measures and logs the item count of specified Lua module tables, useful for monitoring memory growth or detecting table leaks.

Features:

  • Configurable list of Lua modules to measure
  • Configurable measurement interval and recursion depth
  • Scoped to specific APISIX process types (worker, privileged agent)

Changes

  • apisix/plugins/toolset/init.lua — plugin entry point with dynamic sub-plugin load/unload via timer
  • apisix/plugins/toolset/config.lua — default sub-plugin configuration (loaded at runtime, not via schema)
  • apisix/plugins/toolset/src/trace.lua — trace sub-plugin
  • apisix/plugins/toolset/src/table-count/init.lua — table-count sub-plugin
  • apisix/cli/config.lua — default plugin_attr for toolset
  • conf/config.yaml.example — plugin list entry and documented plugin_attr example
  • docs/en/latest/plugins/toolset.md — plugin documentation
  • docs/en/latest/config.json — sidebar navigation entry

@AlinsRan
feat: add toolset plugin
8cc56dc 
The toolset plugin is a diagnostics and observability framework that
hosts multiple lightweight sub-plugins, each independently configured
via plugin_attr and dynamically loaded/unloaded at runtime.

Sub-plugins included:
- trace: instruments APISIX request phases and logs a timing table for
  sampled requests, supports host/path filtering, sampling rate,
  trace header detection, and minimum timespan threshold
- table_count: periodically measures and logs the entry count of
  specified Lua module tables, useful for monitoring memory growth

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. doc Documentation things enhancement New feature or request plugin labels May 26, 2026
AlinsRan and others added 2 commits May 27, 2026 04:15
@AlinsRan
fix: handle timer creation error in toolset plugin init()
2d39d74 
The timer returned by ngx.timer.at() was not checked, causing silent
failure if the timer could not be created (e.g. resource exhaustion).
Apply the same error-checking pattern used in the sync() function.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@AlinsRan
test: add toolset plugin tests and zh docs
be57db2 
- t/plugin/toolset.t: sync loop and config reload tests
- t/plugin/trace.t: phase timing, rate sampling, threshold tests
- t/plugin/trace.host.t: host glob pattern matching tests
- t/plugin/trace.path.t: path glob pattern matching tests
- t/plugin/trace.headers.t: trace header detection and vars tests
- t/plugin/trace.dns.t: DNS resolve phase timing test
- t/plugin/table-count.t: Lua module table counting tests
- t/table-count-example.lua: test fixture for table-count tests
- docs/zh/latest/plugins/toolset.md: Chinese documentation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 27, 2026
@AlinsRan
fix: add license headers, Makefile install entries, and sync race con…
13b8d31 
…dition

- Add Apache 2.0 license headers to all plugin Lua files and test files
- Add toolset plugin install entries to Makefile so luarocks installs
  all plugin files including the src/ and src/table-count/ subdirectories
- Fix race condition in sync(): check stop_timer at function entry to
  prevent a scheduled sync() from re-initializing sub-plugins after
  destroy() has already cleared the cache and set stop_timer = true

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@AlinsRan AlinsRan requested a review from Copilot May 27, 2026 06:33
Copilot AI reviewed May 27, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new global toolset plugin intended as a diagnostics/observability host that can dynamically load/unload lightweight sub-plugins (notably trace and table_count), along with tests, docs, and packaging/config updates.

Changes:

  • Added toolset plugin framework with sub-plugins for request phase timing logs (trace) and periodic Lua table size counting (table_count).
  • Added t::APISIX tests for trace, table_count, and toolset reload/sync behavior.
  • Updated defaults/docs/navigation and installation packaging to include the new plugin and its configuration examples.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 15 comments.

Show a summary per file
File Description
t/table-count-example.lua Test helper module used by table_count tests (tables, circular refs, deep nesting).
t/plugin/trace.t End-to-end tests for trace table output, sampling, and reload behavior.
t/plugin/trace.path.t Tests glob path matching behavior for trace allowlist.
t/plugin/trace.host.t Tests host matching behavior for trace allowlist.
t/plugin/trace.headers.t Tests trace header detection, var capture, and UUID generation.
t/plugin/trace.dns.t Tests DNS resolve phase instrumentation logging.
t/plugin/toolset.t Tests toolset sync loop and reload behavior with config changes.
t/plugin/table-count.t Tests periodic module table counting, circular detection, and depth enforcement.
Makefile Installs toolset plugin Lua files into the runtime Lua dir.
docs/zh/latest/plugins/toolset.md Chinese documentation for toolset plugin configuration and examples.
docs/zh/latest/config.json Adds toolset doc entry to the Chinese docs sidebar/nav.
docs/en/latest/plugins/toolset.md English documentation for toolset plugin configuration and examples.
docs/en/latest/config.json Adds toolset doc entry to the English docs sidebar/nav.
conf/config.yaml.example Adds toolset to plugin list (commented) and documents plugin_attr.toolset examples.
apisix/plugins/toolset/src/trace.lua Implements trace sub-plugin by instrumenting APISIX phases and logging a timing table.
apisix/plugins/toolset/src/table-count/init.lua Implements periodic Lua module table counting with depth/cycle handling and scope filtering.
apisix/plugins/toolset/init.lua Toolset entry point: periodic sync loop and dynamic load/unload of sub-plugins.
apisix/plugins/toolset/config.lua Default sub-plugin configuration values (currently loaded via require).
apisix/cli/config.lua Adds default plugin_attr.toolset configuration in CLI defaults.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugins/toolset/src/trace.lua Outdated
Comment thread apisix/plugins/toolset/src/trace.lua Outdated
Comment thread apisix/plugins/toolset/src/trace.lua
Comment thread apisix/plugins/toolset/src/trace.lua Outdated
Comment thread apisix/plugins/toolset/src/trace.lua
Comment thread t/plugin/trace.t Outdated
Comment thread docs/en/latest/plugins/toolset.md
Comment thread docs/zh/latest/plugins/toolset.md
Comment thread t/table-count-example.lua Outdated
Comment thread t/plugin/trace.t Outdated
AlinsRan and others added 3 commits May 27, 2026 14:41
@AlinsRan
fix(toolset): add sleep after plugin reload in TEST 3 to avoid race c…
8d82723 
…ondition

plugins/reload uses events:post() which is asynchronous - it returns 200
before destroy() is called. Without a sleep, the /hello request can arrive
before trace.destroy() restores the phase hooks, causing spurious trace: logs.

Use ngx.sleep(2) consistent with other reload tests in the test suite.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@AlinsRan
fix(toolset): address code review issues
5537e2f 
- trace: escape regex metacharacters in glob patterns before * substitution
- trace: replace unique_random() pool-depletion logic with math.random(100)
- trace: fix package.loaded reset from false to nil to allow require() reload
- trace: return match_route result to preserve router semantics
- trace: restore dns.resolve in destroy() to prevent stacking on reload
- toolset: change sync log from info to debug to avoid log flooding
- table-count: reset stop=false in init() so plugin can be re-enabled
- table-count: fix depth default from 1 to 10 to match config/docs
- test: fix undefined 'message' variable in trace.t TEST 3
- test: remove dead read-then-overwrite pattern in trace.t TEST 1
- test: fix comment typo in table-count-example.lua
- docs: add canonical link to EN/ZH plugin documentation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@AlinsRan
fix(toolset): correct configuration mechanism in docs and remove inva…
4401f25 
…lid plugin_attr

The toolset plugin reads configuration from apisix/plugins/toolset/config.lua,
not from plugin_attr in config.yaml. The plugin_attr entries added to
cli/config.lua and config.yaml.example were never read by any code.

- Remove plugin_attr.toolset from cli/config.lua
- Remove plugin_attr.toolset example from conf/config.yaml.example
- Rewrite EN/ZH docs to document config.lua as the configuration method
- Restore sync log from debug to info (required by toolset.t TEST 1)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 3 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

doc Documentation things enhancement New feature or request plugin size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AlinsRan