transcription-codeswitching-benchmark

TypeScript tooling for testing speech-to-text APIs on multilingual and code-switching audio.

The repo is set up to run the same audio files through different providers, save the raw responses, map the outputs into one transcript shape, score them against ground truth, and generate JSON/Markdown reports. The goal is reproducible testing for a specific dataset, not a universal provider ranking.

What It Does

• Runs provider adapters against a local audio manifest.
• Saves transcripts, raw provider response paths, timing metadata, and provider errors.
• Scores WER, CER, language ID, language-switch detection, diarization readiness, and latency.
• Reports metrics per provider and per audio file.
• Reports whether each API returned language tags, timestamps, word timestamps, segment timestamps, and speaker labels.
• Keeps provider-specific API code inside src/providers.

What It Does Not Do

• It does not rank providers.
• It does not make broad claims about which provider is better.
• It does not translate or remove code-switched words.
• It does not include real benchmark results.
• It does not include API keys.

Treat results as specific to the files, languages, dialects, speakers, recording setup, provider settings, and model versions used in a run.

Install

npm install

Configure Environment Variables

Copy .env.example to .env or set the variables in your shell.

cp .env.example .env

Credentials are always read from environment variables. Do not put API keys in source files.

OpenAI settings:

• OPENAI_TRANSCRIPTION_MODEL: defaults to gpt-4o-transcribe.
• OPENAI_TRANSCRIPTION_RESPONSE_FORMAT: defaults to json.
• OPENAI_TRANSCRIPTION_LANGUAGE: optional ISO-639-1 language hint, such as en. If this is unset and the manifest has one language, that language is used.
• OPENAI_TRANSCRIPTION_PROMPT: optional transcription prompt.
• OPENAI_TRANSCRIPTION_TIMESTAMPS: used only for verbose_json runs, such as whisper-1 timestamp checks.

Provider model and timestamp notes live in docs/provider-configuration.md.

Add Audio Files

Put audio in data/audio, or point the manifest to files somewhere else. A manifest entry looks like this:

[
  {
    "id": "english-spanish-example-001",
    "audioPath": "./data/audio/english-spanish-example-001.wav",
    "languagePair": ["en", "es"],
    "scenario": "intra-sentential-code-switching",
    "notes": "Example recording with English and Spanish within a single utterance."
  }
]

Keep id stable. It links the audio, ground truth, provider transcript, scores, and report rows.

Create Ground Truth

Create one JSON file per audio item in data/ground-truth. audioId must match the manifest.

{
  "audioId": "english-spanish-example-001",
  "segments": [
    {
      "speaker": "speaker_1",
      "startMs": 0,
      "endMs": 2400,
      "text": "Let's review the roadmap porque necesitamos terminarlo esta semana.",
      "words": [
        { "text": "Let's", "language": "en", "startMs": 0, "endMs": 300 },
        { "text": "review", "language": "en", "startMs": 300, "endMs": 700 },
        { "text": "the", "language": "en", "startMs": 700, "endMs": 850 },
        { "text": "roadmap", "language": "en", "startMs": 850, "endMs": 1200 },
        { "text": "porque", "language": "es", "startMs": 1200, "endMs": 1500 }
      ]
    }
  ]
}

Word-level language tags are needed for language ID and switch detection. Speaker labels are saved too, but diarization scoring still needs a mapping between ground-truth speakers and provider speaker labels.

Run Providers

Run one provider:

npm run benchmark -- --provider openai --manifest ./data/manifest.json

Run every registered provider:

npm run benchmark -- --provider all --manifest ./data/manifest.json

Use --output ./some-directory to write results somewhere other than ./results.

The OpenAI adapter uses the official SDK. The other provider files are adapter stubs: they reserve the provider names, credentials, and result paths, but still need provider-specific SDK or REST code.

Failed provider calls are written to error.json files and to results/errors.json, so one bad file or provider does not stop the whole run.

Score Results

npm run score -- --results ./results --groundTruth ./data/ground-truth

Scores are written beside each transcript as scores.json.

Current metrics:

• WER: edit-distance word error rate over normalized ground-truth and provider text.
• CER: edit-distance character error rate over normalized text.
• Language ID accuracy: compares word-level tags when available, falls back to segment-level tags when comparable, and returns not_available when the provider did not return language metadata.
• Switch detection accuracy: compares language-change points from provider language metadata. It does not infer language from the transcript text.
• Diarization accuracy: records speaker-label availability, but does not score diarization until speaker labels can be mapped.
• Latency: wall-clock request time per audio file.

Generate Reports

npm run report -- --results ./results

This writes:

• results/report.json
• results/report.md

Reports include this disclaimer:

These results are specific to the supplied audio files, languages, dialects, speakers, recording conditions, and provider configurations. They should not be interpreted as universal provider rankings.

The Markdown report has two tables:

• Metrics per provider/audio file.
• API output capabilities: language tags, language-change points, timestamps, word timestamps, segment timestamps, speaker labels, and provider notes.

Normalization

Text normalization is in src/normalization/normalizeTranscript.ts.

Defaults:

• Lowercase text.
• Remove configurable punctuation.
• Preserve apostrophes.
• Normalize whitespace.
• Do not translate text.
• Do not remove code-switched words.

Pass custom NormalizationOptions into metric functions if a run needs different punctuation handling.

Add a Provider Adapter

1. Create src/providers/myProvider.ts.
2. Implement TranscriptionProvider:

import type { TranscriptionProvider } from "./provider.js";

export const myProvider: TranscriptionProvider = {
  name: "my-provider",
  async transcribe(input) {
    // Read credentials from environment variables.
    // Save the raw provider response under input.outputDirectory.
    // Map the response into ProviderTranscriptOutput.
  }
};

3. Add the provider to src/providers/index.ts.
4. Store raw provider responses under results/<provider>/<audio-id>/raw.json.
5. Include model, endpoint, language settings, timestamps requested, and any model/version metadata the provider returns.

Use src/providers/openaiTranscriptionProvider.ts as the working example.

Add a Metric

1. Add a file under src/metrics.
2. Export a small typed function that accepts ground-truth and provider transcript data.
3. Return JSON-safe data. Use not_available when the metric cannot be scored.
4. Add the metric to src/scoreResults.ts.
5. Add it to src/report.ts if it belongs in the Markdown report.

Interpreting Results

Use reports to understand this test set, not to generalize across all speech or all providers. Results can change with:

• Provider API updates.
• Model or version changes.
• Language pair and dialect.
• Speaker accents, overlap, and turn-taking.
• Audio quality, microphone, compression, and background noise.
• Provider options such as diarization, language hints, or custom vocabularies.

Record the run date, provider configuration, model/version, language settings, and preprocessing choices.