SkillSpector-TKV: A lightweight (~144KB) native port in TokenVector with GUI, offline-first scan, and parity rules

[nguyenhungtran18]

Hi NVIDIA SkillSpector team & community,

I developed SkillSpector-TKV, an offline-first, native port of NVIDIA SkillSpector compiled via TokenVector (.tkv) into zero-dependency standalone binaries.

To evaluate whether a compiled native binary brings tangible value to client-side CI/CD gates and local pre-commit hooks, I ran comparative benchmarks against the upstream Python implementation across scan performance, memory footprints, and rule parity.

1. Performance & Resource Benchmarks

• Environment: Intel i7 (12th Gen), Windows 11 / Linux (Mono runtime), NVMe SSD.
• Workload: Static offline scan against a large corpus containing 500+ agent skill files / SKILL.md fixtures (~12MB total code & metadata).

Metric	Python (SkillSpector Upstream)	SkillSpector-TKV (Native)	Delta / Improvement
Cold-Start Overhead	~480 ms (Interpreter init + deps)	~38 ms	~12.6x faster
Throughput (500 skills)	4.25 s	0.82 s	~5.1x faster
Peak Memory (RAM)	~85 MB – 110 MB	~11 MB – 14 MB	~8x reduction
Binary / Artifact Size	~35 MB (Virtualenv / PyInstaller bundle)	~144 KB (Single executable)	Zero Python dependency

Key takeaway: For pre-commit hooks and local desktop use, the sub-50ms latency provides an immediate, zero-friction developer experience without environment setup overhead.

---

2. Rule Parity & Risk Engine Validation

To ensure security guarantees remain strictly identical:

• Parity Status: 100% pass on all 87 upstream detection rules (prompt injection patterns, dangerous OS calls, credential sniffing, MCP tool poisoning).
• Scoring Accuracy: Verified matching the exact 0–100 risk scoring algorithm, severity tiers (SAFE, CAUTION, DO_NOT_INSTALL), and standard CLI exit codes (0, 1, 2).
• Extended Rules (+14): Added specialized detection rules focusing on media-stream exfiltration vectors, obfuscated reverse-shell endpoints, and dynamic URL injection (derived from auditing real-world media download scripts).
• Dual Output: Validated full SARIF JSON schema parity for drop-in GitHub Code Scanning integration.

---

3. Real-World Case Study (Test Fixture)

We tested both engines against real-world repositories featuring heavy network operations and automation logic (such as TokenVector-Media-Downloader):

• Detection Consistency: Both upstream and TKV identified identical network egress and permission elevation patterns with identical final scores (Risk: 35 / CAUTION).
• LLM Semantic Layer: Both implementations successfully parse and preserve DROP ID logic when running optional hybrid LLM checks to prune false positives.

Feedback & Exploration

• Repository: nguyenhungtran18/SkillSpector-TokenVector
• Test fixtures & SARIF comparison artifacts are documented in the repo.

I would love to get feedback from the maintainers on:

1. Edge-case test suites or synthetic datasets you use internally to benchmark false-positive rates.
2. Whether any of the additional 14 detection rules for dynamic script/stream egress would be beneficial to PR into upstream SkillSpector.