Skip to content

llmring/registry

BranchesTags

Repository files navigation

LLMRing Registry

⚠️ Pre-release notice

The pricing, token limits, and capabilities in this registry are under active validation and may be inaccurate. Do not rely on these numbers for production decisions. Always verify against the providers' official documentation.

The official model registry for LLMRing - providing up-to-date pricing, capabilities, and metadata for all major LLM providers.

Overview

The LLMRing Registry is the source of truth for model information across the LLMRing ecosystem. It maintains accurate model data extracted from provider documentation, serving it through GitHub Pages for global, free access.

Key Features:

  • 🤖 Claude-guided extraction - Interactive, high-accuracy model extraction
  • 📦 Versioned JSON files - Historical snapshots with full audit trail
  • 🔄 Smart merging - Preserves existing data when updating
  • 🌐 Served via GitHub Pages at https://llmring.github.io/registry/
  • 🔓 No API keys required for access

Architecture

Registry (This Repo)
├── sources/            # Source documentation (audit trail)
│   ├── anthropic/     # Anthropic model docs (markdown)
│   ├── openai/        # OpenAI model docs
│   └── google/        # Google model docs
├── drafts/            # Pending model updates
│   └── *.draft.json   # Generated by Claude
├── Review & Promotion
│   ├── Diff generation (review-draft)
│   ├── Smart merging (promote)
│   └── Version management
└── Output
    ├── models/        # Current production models
    ├── pages/         # Versioned archives for GitHub Pages
    └── manifest.json  # Registry metadata

Setup

Installation

# Clone the repository
git clone https://github.com/llmring/registry.git
cd registry

# Install with uv (recommended)
uv sync
uv pip install -e .

No API keys or browser automation required! Model extraction is done interactively with Claude.

How to Update Models

The registry uses a Claude-guided interactive workflow for maximum accuracy and simplicity.

Complete Workflow

When providers release new models or change pricing, use the update-registry skill:

1. In Claude Code: "Update the anthropic registry"

2. Claude will guide you through:
   - Save the latest documentation to sources/anthropic/YYYY-MM-DD-models.md
   - Claude reads and extracts models
   - Review the changes
   - Promote to production
   - Commit to git

Manual Step-by-Step

If you prefer to run commands manually:

1. Save Source Documentation

Visit the provider's documentation page and save it as markdown:

# Anthropic
# Visit: https://docs.anthropic.com/en/docs/about-claude/models/overview
# Save to: sources/anthropic/2025-10-20-models.md

# OpenAI
# Visit: https://platform.openai.com/docs/models
# Save to: sources/openai/2025-10-20-models.md

# Google
# Visit: https://ai.google.dev/gemini-api/docs/models
# Save to: sources/google/2025-10-20-models.md

2. Extract Models (with Claude)

Create a draft JSON file in drafts/{provider}.{date}.draft.json by reading the source file and extracting model information following the registry schema.

See .claude/skills/update-registry/skill.md for the complete extraction schema and guidelines.

3. Review Changes

# Compare draft against current production
uv run llmring-registry review-draft --provider anthropic

# This creates drafts/anthropic.YYYY-MM-DD.draft.diff.json showing:
# - Models added
# - Models removed
# - Fields changed

4. Promote to Production

# Promote draft to production
uv run llmring-registry promote --provider anthropic

This will:

  • Merge draft into current registry (preserving existing fields with non-null values)
  • Increment version number
  • Create versioned snapshot at pages/anthropic/v/{N}/
  • Archive source docs
  • Update manifest

5. Commit Changes

git add sources/anthropic/ pages/anthropic/ models/anthropic.json manifest.json
git commit -m "Update Anthropic models - 2025-10-20

Added new models:
- Claude Sonnet 4.5
- Claude Haiku 4.5

Updated all existing models with:
- Added caching support pricing
- Added extended thinking support"
git push

Directory Structure

registry/
├── .claude/
│   └── skills/
│       └── update-registry/   # Claude skill for guided updates
│           └── skill.md
├── sources/                   # Source documents (audit trail)
│   ├── anthropic/
│   │   └── YYYY-MM-DD-models.md
│   ├── openai/
│   └── google/
├── drafts/                    # Extracted model drafts pending review
│   ├── *.draft.json
│   └── *.draft.diff.json
├── models/                    # Current production models (local)
│   ├── anthropic.json
│   ├── openai.json
│   └── google.json
└── pages/                     # Versioned archives for GitHub Pages
    └── {provider}/
        ├── models.json        # Current version
        └── v/                 # Historical versions
            └── {N}/
                ├── models.json
                └── sources/   # Archived source docs

CLI Commands

Core Commands

# Review a draft (generates diff)
uv run llmring-registry review-draft --provider anthropic

# Promote draft to production
uv run llmring-registry promote --provider anthropic

# List available drafts
uv run llmring-registry list-drafts

# Show registry statistics
uv run llmring-registry stats --provider anthropic

Utility Commands

# Normalize draft JSON (clean up formatting)
uv run llmring-registry normalize --provider anthropic

# Export registry data
uv run llmring-registry export --output markdown > models.md
uv run llmring-registry export --output json

Model Schema

Each provider's JSON file contains models in dictionary format with provider:model keys for fast lookup:

{
  "provider": "anthropic",
  "version": 4,
  "updated_at": "2025-10-20T00:00:00Z",
  "extraction_date": "2025-10-20T12:00:00Z",
  "sources": {
    "documents": 1,
    "models_extracted": 10
  },
  "models": {
    "anthropic:claude-sonnet-4-5-20250929": {
      "provider": "anthropic",
      "model_name": "claude-sonnet-4-5-20250929",
      "display_name": "Claude Sonnet 4.5",
      "description": "Our best model for complex agents and coding",
      "model_aliases": ["claude-sonnet-4-5"],

      // Pricing (required)
      "dollars_per_million_tokens_input": 3.0,
      "dollars_per_million_tokens_output": 15.0,

      // Optional pricing fields
      "dollars_per_million_tokens_cache_write_5m": 3.75,
      "dollars_per_million_tokens_cache_write_1h": 6.0,
      "dollars_per_million_tokens_cache_read": 0.3,
      "long_context_threshold_tokens": 200000,

      // Token limits (required)
      "max_input_tokens": 136000,
      "max_output_tokens": 64000,

      // Capabilities (boolean)
      "supports_vision": true,
      "supports_function_calling": true,
      "supports_json_mode": true,
      "supports_parallel_tool_calls": true,
      "supports_streaming": true,
      "supports_caching": true,
      "supports_thinking": true,
      "is_active": true,

      // Metadata
      "api_endpoint": "chat",
      "tool_call_format": "anthropic",
      "max_temperature": 1.0,
      "min_temperature": 0.0
    }
  }
}

Key Design Decisions

Why Claude-Guided Extraction?

The previous system used an automated LLM pipeline with multi-document voting and validation. This was over-engineered for a task that happens ~once per month.

New approach benefits:

  • ✅ More accurate - Claude can reason about edge cases interactively
  • ✅ Simpler - 65% less code to maintain
  • ✅ Faster - No pipeline overhead or retries
  • ✅ Debuggable - You see the extraction happen in real-time
  • ✅ Interactive - Can ask questions and verify as you go

Smart Merge Logic

When promoting drafts, the system:

  • ✅ Updates fields where draft has non-null values
  • ✅ Preserves production fields when draft has null
  • ✅ Prevents accidental data loss
  • ✅ Maintains backward compatibility

Example: If production has api_endpoint: "chat" and draft has api_endpoint: null, the production value is preserved.

Troubleshooting

Common Issues

Problem: Can't find latest draft

# Solution: Check drafts directory
ls -lt drafts/*.draft.json

# Or use list-drafts command
uv run llmring-registry list-drafts

Problem: Promote fails with validation error

# Solution: Check the error message carefully
# Common issues:
# - Missing required fields (model_name, pricing)
# - Invalid JSON syntax
# - Wrong data types

# Manually edit the draft JSON and re-run promote

Problem: Changes not showing up

# Solution: Check if draft was actually different from production
uv run llmring-registry review-draft --provider anthropic

# If no changes, promote will skip

Integration with LLMRing

The registry serves as the data source for the entire LLMRing ecosystem:

  1. Static Hosting: JSON files served via GitHub Pages
  2. Registry URL: https://llmring.github.io/registry/
  3. Manifest: Contains version info and provider index
  4. Updates: On-demand via Claude-guided workflow

Client usage:

from llmring import LLMRing

# Automatically fetches latest registry
ring = LLMRing()

# Get available models
models = ring.get_available_models()

Contributing

We welcome contributions! The simplest way to contribute is to update model data when providers release changes.

Adding a New Provider

  1. Create directory: sources/{provider}/
  2. Save documentation to markdown
  3. Use Claude to extract models following the schema
  4. Review, promote, and commit

License

MIT License - see LICENSE for details.

Links

Built with ❤️ by the LLMRing team

About

No description, website, or topics provided.

Resources

Readme

License

MIT license

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages