Introducing the AI Skill Registry Validation Framework (Security, Scoring & Drift Detection)

[fkauanGIT]

What are you sharing?

Custom skill

Name or short title

AI Skill Registry Validation Framework

What did you build or discover?

I recently contributed an AI Skill Registry Validation Framework designed to improve security, quality control, and maintainability across the repository's growing ecosystem of AI skills.

As the project scales beyond 1,500 skills, manually reviewing every contribution becomes increasingly difficult. This framework introduces automated validation tooling that helps maintainers identify security risks, documentation issues, metadata quality problems, and unexpected modifications before they reach production.

The contribution includes:

• Security scanning for dangerous patterns such as curl|bash, Invoke-Expression, dynamic eval(), hardcoded credentials, and fork bombs
• Skill quality scoring (0-100)
• Drift detection using file hashes
• Registry-wide health reporting
• Python tooling with automated tests

The goal was to provide maintainers with practical tooling that improves review quality while remaining completely backward compatible with existing skills and workflows.

How it works

The framework consists of four main components:

Security Scanner

Scans skills for potentially dangerous patterns and reports warnings or errors.

Examples:

• Remote code execution via curl | bash
• PowerShell Invoke-Expression
• Dynamic eval()
• Hardcoded secrets
• Process spawning abuse

Skill Scoring

Evaluates each skill and generates a score from 0 to 100 based on:

• Metadata completeness
• Documentation quality
• Security indicators
• Overall maintainability

Drift Detection

Uses hashes to identify unexpected modifications and helps maintainers understand when a skill has changed since the last approved baseline.

Registry Reports

Generates consolidated health reports for the entire registry.

Example commands:

npm run security:scan
npm run score:skills
npm run drift:check
npm run registry:report

### Links, screenshots, or references


---

### Links, screenshots, or references

```markdown
- Repository:
https://github.com/fkauanGIT/antigravity-awesome-skills

- Pull Request: 
https://github.com/sickn33/antigravity-awesome-skills/pull/718

- Example Output:

🔐 Security Scan — 1569 skills scanned
Errors  : 35
Warnings: 29

- Branch:
feat/skill-registry-validation-framework

### Before posting

- [X] I removed secrets, credentials, and private data from anything I shared.

[fkauanGIT]

I'd love feedback from maintainers and contributors on a few areas:

• Additional security patterns worth detecting
• False-positive handling strategies
• Severity levels (Info / Warning / Error / Critical)
• SARIF export for GitHub Code Scanning
• Additional registry health metrics

I'm especially interested in ideas that would help scale review quality as the skill ecosystem continues to grow.

[sickn33]

Thanks for opening this up, and thanks again for keeping the final PR Python-only and aligned with the existing tooling.

A few maintainer-side thoughts on where I think the next iterations would be most useful:

1. False-positive handling should stay explicit and auditable. The current security-allowlist marker is a good start. I would avoid broad repo-level suppressions for now; a per-line marker with a short reason keeps review context close to the risky command. If this grows, the next useful step would be reporting allowlisted findings separately so maintainers can see what was intentionally suppressed.

2. Severity should probably remain small at first. The scanner already has error, warning, and info in the model. I would not rush into a large severity taxonomy unless we wire it to clear behavior. A practical mapping could be:

   • error: likely dangerous if copied/executed, or likely leaked secret
   • warning: context-dependent risk, needs maintainer review
   • info: hygiene or advisory signal only

3. SARIF export is the most useful integration follow-up. If someone wants to extend this, I would suggest a small PR that only adds --sarif to the security scanner first. Keep it scoped to scanner findings, use stable rule IDs like SEC002, and map locations back to skills/<id>/SKILL.md line numbers. That would make GitHub Code Scanning useful without coupling it to scoring or drift yet.

4. Additional patterns worth considering:

   • bash -c / sh -c fed from variables or downloaded content
   • python -c, node -e, perl -e, ruby -e when combined with network/download examples
   • token-looking strings beyond api_key=..., especially common prefixes like ghp_, sk-, xoxb-, AKIA, etc.
   • destructive filesystem commands outside root, for example rm -rf ~/.ssh, rm -rf $HOME, or find ... -delete
   • credential exfiltration patterns such as piping env output to curl/webhooks

5. Registry health metrics should be few and actionable. The ones I would find useful are: count of skills with security errors, count of allowlisted findings, lowest-scoring skills, skills missing required/recommended sections, and drift summary since the last reviewed baseline. I would avoid adding metrics that do not tell a maintainer what to review next.

My preference would be to keep follow-ups as small separate PRs: first SARIF for the scanner, then severity/allowlist reporting refinements, then any new patterns with tests. That keeps the blast radius easy to review.