AISBench Testing v1.1

A generic AI skill for running AISBench benchmarks from a container. Covers dataset management, config creation, precision/performance test execution, and AISBench reinstallation.

Features

• Server-agnostic — All vLLM service and container info managed via config file. Change config to switch targets.
• Container-agnostic — Any container from a vllm-ascend image works. AI guides you through creation if needed.
• AI-tool-agnostic — Pure Markdown + bash scripts. Works with Claude Code, TRAE, or any AI tool.
• Prompt-first, scripts-as-supplement — AI handles flexible scenarios (config generation, command orchestration); scripts cover high-frequency operations.
• Self-installing — SKILL.md auto-completes installation and configuration on first activation.

Prerequisites

No environment dependencies for installation itself. Before running benchmarks, you need:

1. Target vLLM service running — Confirm service IP and port are reachable
2. A container from a vllm-ascend image — For installing and running AISBench. AI will guide you to create one if you don't have it yet.
3. Datasets ready — Test datasets downloaded to the container's designated directory.

AI will guide you through setup on first activation.

Quick Start

1. Install

Recommended: Let AI install for you

Clone this repo first, then ask AI to read the local directory and install:

I'm using Claude Code (or another tool). Please read /path/to/aisbench-testing/ and install this skill correctly. Tell me how to use it once installed.

Replace the path with your actual path. AI will:

1. Read SKILL.md to understand the skill
2. Auto-detect the skill directory for your tool
3. Complete installation
4. Guide you through first-time configuration (including container discovery/creation)

Alternative: Manual install

Tool	Skills Directory
Claude Code	~/.claude/skills/aisbench-testing/SKILL.md
OpenClaw	Check tool docs for skills config path
Codex CLI	Check tool docs for skills config path
Other tools	Place SKILL.md in the tool's skills directory

2. Usage

Two ways to activate:

Auto-trigger — Chat normally. AI matches this skill when keywords like "benchmark", "precision test", "AISBench", "GPQA", "AIME" are mentioned (Chinese: 打流, 精度测试, 性能测试).

Explicit (Claude Code) — Use /aisbench-testing after installation.

On first activation, AI will:

1. Install helper scripts
2. Configure server and container info (guide container creation if needed)
3. Check AISBench installation status

3. Examples

You: Run precision test, model kimi_k26, dataset gpqa
AI: Please confirm:
    - vLLM service: 192.168.13.153:8000
    - Model: kimi_k26
    - Dataset: gpqa
    - Max output length: 98304
    - Mode: all
    - Number of prompts: all
    Confirm?
You: Confirmed
AI: Generating config... OK
    Starting test... [running]
    ...
    Test complete
    GPQA accuracy: 85.3%

You: I don't have an AISBench container, help me create one
AI: Sure. First, do you have a vllm-ascend Docker image? What's the image ID/name?
You: Image ID is abc123
AI: What name for the container?
You: Let's call it aisbench_test
AI: Creating container... OK
    Installing AISBench... OK
    Container aisbench_test is ready. Config saved.

Configuration

Config file: ~/.aisbench_testing/config.json

AI guides you through configuration on first activation. You can always tell AI to update:

• "Change vLLM service IP to 10.0.0.5"
• "Switch default model to deepseek_v3"
• "Use a different AISBench container"

See SKILL.md for detailed config field descriptions.

Repo Structure

aisbench-testing/
├── SKILL.md              # Core skill file (AI entry point)
├── README.md             # This file
├── LICENSE               # MIT
├── config.json           # Config template
└── scripts/              # Helper scripts
    └── init-config.sh    # Initialize config

License

MIT