Skip to content

neerajkumar-n/IQ

BranchesTags

Repository files navigation

AI Share of Voice (AI SOV) Analyzer

A production-quality CLI tool for B2B companies to measure how often their brand appears in AI model responses across different buyer personas and prompt intents — and to compute actionable Share of Voice metrics.

Table of Contents

Overview

AI SOV Analyzer answers the question: "When buyers ask AI models about our category, does our company show up?"

It runs a configurable set of prompts against any AI model endpoint, parses the responses for brand mentions, and computes:

  • Presence Rate — how often your company appears
  • AI Share of Voice — your mention share vs competitors
  • Recommendation Rate — how often you're listed first
  • Missed Prompts — high-intent queries where competitors appear but you don't

Works with any OpenAI-compatible endpoint — bring your own API key and model URL. No lock-in to any specific provider.

Project Structure

/
├── config.json            # All variables: company, models, API keys, settings
├── prompts.json           # Prompts with persona + intent tags
├── runner.py              # CLI entry point
├── parser.py              # Response text parsing & mention extraction
├── metrics.py             # SOV metrics computation engine
├── insights.py            # Rule-based insight generation
├── utils.py               # Config loading, env var expansion, logging
├── model_clients/
│   ├── __init__.py        # Provider registry (auto-falls back to generic)
│   ├── generic_client.py  # Universal OpenAI-compatible client (default)
│   ├── openai_client.py   # Native OpenAI client (optional)
│   └── anthropic_client.py# Native Anthropic client (optional)
├── output.json            # Generated: structured metrics + insights
└── raw_responses.json     # Generated: full responses + parsed mentions

Setup

1. Install dependencies

pip install openai

Only the openai package is required — the generic client uses it to talk to any OpenAI-compatible endpoint. If you use the native Anthropic client specifically, also run pip install anthropic.

2. Set API keys as environment variables (if needed)

export MY_API_KEY="your-key-here"

API keys in config.json use the ${VAR_NAME} placeholder syntax and are resolved at runtime. Never hardcode keys directly in config.json.

No API key? If you're using a local model via Ollama, skip this step entirely — local models require no authentication. See Free and Zero-Cost Options.

3. Configure your company and models

Edit config.json with your company name, competitors, and at least one model endpoint (see Configuration below).

Configuration

config.json is the single source of truth for all runtime variables.

Company fields

Field Type Description
company string Your company's primary name
aliases string[] Alternate names / abbreviations to also detect
competitors string[] Competitor names to track

Settings fields

Field Type Description
settings.runs_per_prompt int How many times to run each prompt per model
settings.temperature float Model temperature (0.0–1.0)
settings.max_tokens int Max tokens per response

Model fields

Field Required Description
name yes Display name shown in output
endpoint yes Base URL of the API, e.g. https://openrouter.ai/api/v1
api_key no* ${ENV_VAR} reference to your key. Can be omitted entirely for local models (e.g. Ollama) that require no authentication.
model yes Model ID as the endpoint expects it
provider no Hint for client selection. Omit or set to "generic" for any custom endpoint. Use "openai" or "anthropic" only if you want the native SDK behaviour.
extra_headers no Additional HTTP headers (e.g. required by OpenRouter)

* api_key is required for cloud-hosted services (Groq, Gemini, OpenRouter, OpenAI, etc.) but can be omitted for local models that have no authentication.

Minimal model entry

{
  "name": "my-model",
  "endpoint": "${MY_ENDPOINT}",
  "api_key": "${MY_API_KEY}",
  "model": "my-model-id"
}

Full example with multiple endpoints

{
  "company": "Your Company",
  "aliases": ["YourCo", "Your Co"],
  "competitors": ["Competitor A", "Competitor B", "Competitor C"],
  "settings": {
    "runs_per_prompt": 1,
    "temperature": 0.7,
    "max_tokens": 1024
  },
  "models": [
    {
      "name": "my-primary-model",
      "provider": "generic",
      "api_key": "${MY_API_KEY}",
      "model": "${MY_MODEL_ID}",
      "endpoint": "${MY_ENDPOINT}"
    },
    {
      "name": "my-second-model",
      "provider": "generic",
      "api_key": "${MY_SECOND_API_KEY}",
      "model": "${MY_SECOND_MODEL_ID}",
      "endpoint": "${MY_SECOND_ENDPOINT}",
      "extra_headers": { "HTTP-Referer": "https://your-site.com" }
    }
  ]
}

OpenRouter (openrouter.ai) is a convenient way to access GPT-4, Claude, Gemini, Perplexity Sonar, Llama, Mistral and many others under a single API key and a single OpenAI-compatible interface.

model_examples (copy-paste reference)

config.json includes a model_examples key alongside models. These entries are never executed — the runner only reads the models array. They exist purely as ready-to-use snippets. Copy any entry into "models" to activate it.

Free and Zero-Cost Options

You do not need to pay for API access to run this tool. The options below are either entirely free or have a generous free tier.

Option 1 — Ollama (local, completely free, no API key)

Run open-source models on your own machine. No account, no API key, no cost.

Install Ollama: https://ollama.com

# pull a model once
ollama pull llama3

# start the server (runs on http://localhost:11434)
ollama serve

Add to config.json — note there is no api_key field:

{
  "name": "ollama-llama3",
  "provider": "ollama",
  "model": "llama3",
  "endpoint": "http://localhost:11434/v1"
}

Trade-off: Ollama models are open-source (Llama, Mistral, etc.), not the same as ChatGPT or Claude. Results reflect what those models say, not what commercial AI assistants say.

Option 2 — Groq (free tier, API key required)

Groq provides a free tier with generous rate limits. Sign up at https://console.groq.com.

export GROQ_API_KEY="your-groq-key"
{
  "name": "groq-llama3",
  "provider": "groq",
  "api_key": "${GROQ_API_KEY}",
  "model": "llama3-8b-8192",
  "endpoint": "https://api.groq.com/openai/v1"
}

Option 3 — Gemini Flash (free tier, API key required)

Google's Gemini Flash has a free tier via AI Studio. Get a key at https://aistudio.google.com.

export GEMINI_API_KEY="your-gemini-key"
{
  "name": "gemini-flash",
  "provider": "gemini",
  "api_key": "${GEMINI_API_KEY}",
  "model": "gemini-1.5-flash",
  "endpoint": "https://generativelanguage.googleapis.com/v1beta/openai"
}

Option 4 — OpenRouter (aggregator, some free models)

OpenRouter gives access to many models under one API key. Some models are free. Sign up at https://openrouter.ai.

export OPENROUTER_API_KEY="your-openrouter-key"
{
  "name": "openrouter-mixtral",
  "provider": "openrouter",
  "api_key": "${OPENROUTER_API_KEY}",
  "model": "mistralai/mixtral-8x7b-instruct",
  "endpoint": "https://openrouter.ai/api/v1"
}

Cost reference for paid APIs

If you do use paid APIs, a full SOV run is inexpensive:

Service Typical cost per full run (10 prompts × 5 models)
OpenAI GPT-4o ~$0.10–$0.30
Anthropic Claude ~$0.10–$0.30
Perplexity Sonar ~$0.05–$0.20
Groq / Gemini Flash Free (within tier limits)
Ollama (local) $0.00

A weekly automated run across all major models costs roughly $5–$15/month.

Prompts

prompts.json is an array of prompt objects:

[
  {
    "id": "disc-001",
    "persona": "CTO",
    "intent": "discovery",
    "prompt": "What are the best B2B payment platforms for a SaaS company?"
  }
]
Field Values Description
id string Unique identifier for tracking
persona CTO, Engineer, Finance, Founder, ... Buyer role
intent discovery, comparison, decision, integration Funnel stage
prompt string The exact question sent to the AI model

Running the Tool

Basic run

python runner.py

Custom config / prompts paths

python runner.py --config path/to/config.json --prompts path/to/prompts.json

Custom output directory

python runner.py --output-dir results/run-$(date +%Y%m%d)/

Validate config without calling any APIs

python runner.py --dry-run

All options

python runner.py [-h]
                 [--config CONFIG]
                 [--prompts PROMPTS]
                 [--output-dir OUTPUT_DIR]
                 [--log-level {DEBUG,INFO,WARNING,ERROR}]
                 [--dry-run]

Output Files

output.json

Structured metrics and insights:

{
  "meta": {
    "company": "Your Company",
    "run_timestamp": "2024-01-15T10:30:00+00:00",
    "total_results": 40,
    "models_used": ["my-primary-model", "my-second-model"],
    "prompts_count": 20
  },
  "summary": {
    "total_prompts_run": 40,
    "company_presence_rate": 0.425,
    "ai_share_of_voice": 0.183,
    "recommendation_rate": 0.075,
    "company_mention_count": 22,
    "competitor_mention_count": 98,
    "competitor_breakdown": {
      "Competitor A": { "presence_rate": 0.875, "mention_count": 51 }
    }
  },
  "by_model": { "...per model breakdown..." },
  "by_persona": { "...per persona breakdown..." },
  "missed_prompts": [ "...prompts where competitors appear but company does not..." ],
  "insights": [
    "Your Company has moderate AI visibility (42.5% presence rate).",
    "Competitor 'Competitor A' dominates with a 87.5% presence rate.",
    "Your Company is absent from 12 prompts where competitors are mentioned."
  ]
}

raw_responses.json

Full responses with parsed mention data per call:

[
  {
    "model": "my-custom-model",
    "persona": "CTO",
    "intent": "discovery",
    "prompt_id": "disc-001",
    "prompt": "What are the best B2B payment platforms...",
    "run": 1,
    "response": "Here are the top platforms: 1. ...",
    "error": null,
    "parsed_mentions": {
      "company_mentioned": false,
      "competitors_mentioned": ["Competitor A", "Competitor B"],
      "rank": null,
      "mention_count": 0
    }
  }
]

How to Add Models

Any endpoint that speaks the OpenAI Chat Completions format works with zero code changes — just add an entry to config.json:

{
  "name": "any-display-name",
  "endpoint": "https://your-endpoint.com/v1",
  "api_key": "${YOUR_ENV_VAR}",
  "model": "model-id-as-your-endpoint-expects"
}

The provider field defaults to "generic" when omitted. The generic client handles retry logic, temperature, and max_tokens automatically.

If your endpoint is NOT OpenAI-compatible

  1. Create model_clients/myprovider_client.py implementing:
def run_prompt(model_config: dict, prompt: str, system_prompt: str, settings: dict) -> str | None:
    # call your API, return response text or None on failure
    ...
  1. Register it in model_clients/__init__.py:
from model_clients.myprovider_client import run_prompt as myprovider_run

PROVIDER_MAP = {
    ...
    "myprovider": myprovider_run,
}
  1. Set "provider": "myprovider" in your config.json model entry.

How to Add Prompts

Add entries to prompts.json — no code changes required:

{
  "id": "dec-006",
  "persona": "VP Sales",
  "intent": "decision",
  "prompt": "What CRM tools do you recommend for enterprise B2B sales teams?"
}

Intent values: discovery | comparison | decision | integration

Persona values: any string matching your buyer personas.

Metrics Reference

Metric Formula Interpretation
Presence Rate prompts with company / total prompts Visibility breadth
AI Share of Voice company mentions / (company + all competitor mentions) Competitive share
Recommendation Rate times ranked #1 / total prompts Top-of-mind strength
Missed Prompts competitor present, company absent Visibility gaps

Example Output

============================================================
  Results for: Your Company
============================================================
  Prompts run:         40
  Presence Rate:       42.5%
  AI Share of Voice:   18.3%
  Recommendation Rate: 7.5%
  Missed Prompts:      12
============================================================

Key Insights:
  • Your Company has moderate AI visibility (42.5% presence rate). There is clear room for improvement.
  • Low recommendation rate (7.5%) despite moderate visibility — Your Company is mentioned but rarely positioned first.
  • Significant persona gap detected: strong presence in 'CTO' (70.0%) but weak in 'Engineer' (20.0%).
  • Competitor 'Competitor A' dominates with a 87.5% presence rate — significantly outpacing Your Company.
  • Your Company is absent from 12 prompt(s) where competitors are mentioned. These are high-priority visibility gaps.

Full results saved to: output.json
Raw responses saved to: raw_responses.json

License

This project is licensed under the Apache License 2.0.

About

No description, website, or topics provided.

Resources

Readme

License

View license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages