defect-check

defect-check is a standalone, framework-free inspection engine for AI artifacts. It accepts Skills, Tools, and Prompts as input, runs multi-dimensional quality checks, and returns structured defect reports with scores and severity ratings.

• 🔍 Four inspection modules — QDS, QDT, QDP, and Cross (PS/PT/ST)
• 🤖 LLM-powered analysis — supports OpenAI, Anthropic, and DashScope providers
• 🎚️ Three inspection levels — L1 (quick), L2 (standard), L3 (deep)
• 📦 Zero infrastructure — no database, no API server, no task queue
• 🏗️ Framework-free — bring your own runtime, the engine stays pure

• defect-check
  • Table of Contents
  • Installation
  • Quick Start
  • Inspection Levels
  • LLM Configuration
  • Inspection Modules
  • Response Format
    • Defect Fields
  • API Reference
    • defect_check.check(...)
    • defect_check.check_single(...)
    • defect_check.check_cross(...)
    • Exported Types
  • Development
  • Contributing
  • License

pip install defect-check

Requires Python ≥ 3.11

import asyncio
import defect_check

async def main():
    result = await defect_check.check(
        tools=[
            {
                "name": "lookup_order",
                "description": "Query orders by order ID",
                "parameters": {"type": "object"},
            }
        ],
        prompts=[
            {"name": "system", "content": "You are an order assistant."}
        ],
        skills=[
            {"id": "orders", "name": "orders", "content": "# Orders workflow"}
        ],
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
    )
    print(result)

asyncio.run(main())

Use check_level to control inspection depth:

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    check_level="L3",
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)

When omitted, QDS determines the level using the bundled checklist, while QDT, QDP, and Cross determine it via the LLM. You can also pass check_level through options:

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(check_level="L2"),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)

The legacy qdp_check_level option remains supported for backwards compatibility. Conflicting values (e.g. check_level="L3" + options={"qdp_check_level": "L1"}) will raise an error.

LLM settings are passed explicitly by the caller — the package does not read .env files or environment variables for LLM configuration.

You can also pass a pre-configured client object via the provider parameter, bypassing the four llm_* parameters:

from defect_check.llm import DefectCheckTextClient

# Build your own client, then pass it in
client = DefectCheckTextClient(my_custom_provider)

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    provider=client,
)

The package provides four inspection modules, each targeting a different artifact dimension:

Every supplied Skill, Tool, and Prompt is inspected. The response always uses a consistent envelope — results is always a list: one input produces one result item, multiple inputs produce multiple result items.

Inspection rules and prompt templates are packaged in the wheel. QDT, QDP, and Cross load YAML resources; QDS loads the bundled checklist.py.

{
  "schema_version": "1.0",
  "status": "completed",
  "results": [
    {
      "module": "QDT",
      "check_type": "artifact",
      "status": "completed",
      "check_level": "L2",
      "artifacts": [{"type": "tool", "id": "lookup_order", "name": "lookup_order"}],
      "score": {"total_score": 100.0, "max_score": 100.0, "grade": null, "gate_result": "PASS"},
      "defect_summary": {"total_defects": 0, "p0_count": 0, "p1_count": 0, "p2_count": 0},
      "defects": [],
      "error": null,
      "details": {},
      "metadata": {}
    }
  ],
  "summary": {
    "total_results": 1,
    "completed_results": 1,
    "failed_results": 0,
    "skipped_results": 0,
    "total_defects": 0,
    "p0_count": 0,
    "p1_count": 0,
    "p2_count": 0,
    "gate_result": "PASS"
  },
  "errors": [],
  "metadata": {"execution_time_seconds": 0.0}
}

Each defect in the defects list contains these canonical fields:

Inspect caller-provided Skills, Tools, and Prompts.

async def check(
    tools: list[dict] | None,
    prompts: list[dict] | None,
    skills: list[dict] | None,
    *,
    check_level: str | None = None,
    options: DefectCheckOptions | dict | None = None,
    provider: Any | None = None,
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
) -> dict[str, Any]

Inspect a single artifact. See the API documentation for details.

Run cross-artifact inspection (PS/PT/ST). See the API documentation for details.

from defect_check import (
    DefectCheckOptions,
    DefectCheckResponse,
    DefectItem,
    DefectSummary,
    InspectionResult,
    InspectionError,
    ScoreResult,
    ResponseSummary,
    ArtifactReference,
    SkillArtifact,
    PromptArtifact,
)

# Clone the repository
git clone https://github.com/sanityops-org/artifact-defect-check.git
cd artifact-defect-check

# Create a virtual environment
python -m venv .venv && source .venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Please make sure to update tests as appropriate and adhere to the existing code style.

This project is licensed under the Apache License 2.0 — see the LICENSE file for details.