llmw2 – Next-generation LLM API Wrapper

llmw2 adds a provider-agnostic execution pipeline for running large prompt sets across OpenAI, OpenRouter, and local OpenAI-compatible servers. It keeps on-disk caching, exposes the OpenAI Batch API, and supports multi-modal (vision) payloads while remaining simple to call from any Python project.

Installation

pip install -r requirements.txt

Set credentials via environment variables or when instantiating LLMClient:

• OPENAI (or LLMW2_OPENAI) for OpenAI keys
• OPENROUTER for OpenRouter keys
• LOCAL for local server tokens (if required)

Quick Start

from llmw2 import LLMClient

client = LLMClient()
result = client.chat(
    "gpt-4o",
    prompt="Summarize the repo architecture in one paragraph.",
)
print(result.response)

Mass Runs with Caching

prompts = [f"Question #{i}: ..." for i in range(100)]
results = LLMClient().run_many(
    "gpt-4o",
    prompts,
    concurrency=40,
)

Caching is stored in ~/.cache/llmw2 by default; repeated calls with identical payloads reuse prior responses instantly.

OpenAI Batch API

client = LLMClient()
submission = client.batch.create_sync("gpt-4o", prompts)
status = client.batch.status_sync("gpt-4o", submission.batch_id)

Vision-Language Requests

from llmw2 import Message, MessagePart

client = LLMClient()
resp = client.chat(
    "local-qwen2.5vl",
    messages=[
        Message.from_text("system", "You describe charts."),
        Message(
            role="user",
            content=[
                MessagePart(type="image_url", image_url="https://example.com/chart.png"),
                MessagePart(type="text", text="Explain the key trend."),
            ],
        ),
    ],
)

Local OpenAI-Compatible Servers

Add a model entry in llmw2/models.yaml that points to your server:

my-local-model:
  provider: openai_compatible
  api_key: local
  model: my-model-name
  base_url: http://localhost:8000/v1
  capabilities: [chat]

Then call it like any other model: LLMClient().chat("my-local-model", prompt="...").

Design Notes

Detailed design decisions and component breakdown live in docs/llmw2_design.md.