AutoEIT

Scoring Elicited Imitation Tasks by hand is slow and raters disagree most exactly where the 0–4 judgement matters; this automates transcription and scoring, and measures how far the automation can be trusted.

Results

Agreement between two independent scorers built against the same rubric — one deterministic, one an LLM — over 120 items from 4 learners in a provided sample dataset.

Metric	Value	Reading
Quadratic-weighted κ	0.71	Substantial agreement
Exact agreement	50%	Same score on half of items
Within 1 point	86%	Rarely more than one band apart
Mean absolute error	0.64	Typical gap, in rubric points
Mean bias (LLM − rule)	−0.54	The LLM is systematically stricter

Confusion matrix (rows = rule-based, columns = LLM):

         0    1    2    3    4
  0      4    1    0    0    0
  1      8   14    0    0    0
  2      5   20   17    3    1
  3      0    8    7    7    0
  4      0    0    3    4   18

Nearly all the mass sits below the diagonal: the rule-based scorer consistently awards a higher band. It concentrates at the 2/3 boundary — the same judgement the rubric itself flags as ambiguous. The two methods are not making random errors relative to each other; they draw one specific line in different places.

κ = 0.71 is agreement, not accuracy. Both scorers could be wrong together without moving it. See Evaluation for what was measured and Limitations for what it does not establish.

No participant data, identifiers, or instrument content are in this repository. Everything here runs on synthetic examples.

How it works

• Transcribe — Whisper with anti-hallucination decode settings that are load-bearing, not defaults: EIT recordings contain long silences, and an unconstrained model fills them with repeated tokens or another language.
• Align — dynamic-programming alignment maps flat audio segments onto numbered items in administration order, so a segment recorded minutes later cannot be captured by item 1. Near-verbatim matches are flagged rather than dropped, because the prompt and the learner's imitation are both on the tape.
• Score (rules) — fuzzy content-word overlap, edit distance, and response length through an explicit threshold cascade. Deterministic and free.
• Score (LLM) — the same rubric as a system prompt, one call per item at temperature 0, so the two can be compared against each other.

Preflight runs before anything expensive — a multi-gigabyte model load, or paid API calls — and exits non-zero on missing files, a malformed or truncated instrument, or an absent API key, reporting every problem at once.

Quickstart

Install

git clone <this-repo> && cd autoeit
python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

Transcription additionally needs pip install -r requirements-asr.txt (large: torch alone is over 2 GB). Scoring and evaluation do not.

Configure

The instrument is not committed. Start from the synthetic example:

cp config/stimuli.example.json config/stimuli.json

For LLM scoring only, cp .env.example .env and set OPENAI_API_KEY.

Run

python scripts/run_scoring.py examples/synthetic_responses.csv

Expected output — two synthetic learners, correctly separated by proficiency:

PREFLIGHT (scoring)
  ok stimuli loaded: 30 items
PREFLIGHT PASSED
demo-a: mean 3.50  distribution 0=0 1=0 2=5 3=5 4=20
demo-b: mean 1.57  distribution 0=4 1=8 2=16 3=1 4=1
Wrote 60 rows to output/rule_scores.csv

If those two means are 3.50 and 1.57, the install is correct.

The other two entry points:

python scripts/run_scoring_llm.py responses.csv --limit 5   # cheap smoke test
python scripts/run_transcription.py                         # needs ASR extras

Input is a CSV with participant,item,response — see examples/synthetic_responses.csv.

Evaluation

What was measured. Inter-method agreement between the rule-based scorer and gpt-4o-mini at temperature 0, both applying the Ortega et al. (2002) meaning-based rubric to the same human transcriptions. 30 items × 4 learners = 120 paired scores, from a provided sample dataset rather than data I collected. Zero API errors in the run, so no items were dropped.

How. Each scorer produced a 0–4 score per item independently; neither saw the other's output. Metrics come from evaluation/agreement.py, which computes Cohen's κ with quadratic weights, exact and within-1 agreement, MAE, and directional bias. Quadratic weighting is the headline because plain percent agreement flatters an ordinal scale — it treats a 3-vs-4 disagreement like a 0-vs-4 one — and does not correct for chance.

Reproduce it on any two score columns:

python -m evaluation.agreement scores.csv --a rule_score --b llm_score

The underlying learner data cannot be published, so the table above is not reproducible from this repository. The metric code is, and is unit-tested against hand-computed cases.

What this is not. No human reference ratings were available. This measures whether two automated methods agree, not whether either is right. The rule-based thresholds have never been calibrated against a human rater. Treat κ = 0.71 as a consistency ceiling. With 4 learners from a single sample dataset, no broad conclusion follows.

Verify the repository itself:

pytest                                   # 76 passed, 1 xfailed
python -m autoeit.preflight --self-test  # 7/7

The suite needs no data, no network, and no credentials. The single xfail is the morphology limitation below, and is strict — it fails if it starts passing without the note being updated.

Limitations

Stimulus and learner cannot be told apart. Each recording contains the prompt and the imitation, and no text-level method separates them. A verbatim transcription is genuinely ambiguous. Items at ≥0.95 similarity are flagged probable_stimulus_echo rather than dropped — in the sample dataset above this reached 20 of 30 items for one participant. Do not score transcription output without reviewing those flags.

The scorer is weakest where the instrument is most informative. Spanish inflectional morphology sits right on the matching threshold: llegues/llegas is 0.71 similar against a 0.75 cutoff, so a subjunctive-to-indicative substitution reads as a missing word and costs a band it should not — and that substitution is exactly what an EIT is built to elicit. Pinned as a strict xfail in tests/test_score_rules.py. Fixing it needs morphology-aware matching or calibration, not a threshold nudge.

The overlap metrics measure recall, not precision. They ask what fraction of the target appears in the response, with no penalty for anything else present. A response containing the target verbatim scores at least 3 regardless of padding — so a merged prompt-plus-learner segment, or a Whisper repetition hallucination, arrives looking like a solid 3.

Thresholds are uncalibrated. Every number in the cascade was set by hand against a small development sample.

ASR degrades at low proficiency. The most fragmentary speech — where an accurate score matters most — is where transcription is least reliable, and it fails plausibly rather than loudly.

The rubric exists in two forms. Numeric thresholds in score_rules.py and prose in RUBRIC_PROMPT. Changing one is not changing the other.

Data handling

EIT recordings are identifiable speech. Publication safety is mechanical, not habitual: recordings, transcripts, and results are gitignored by pattern; the instrument loads from a gitignored config/stimuli.json with only a synthetic example committed; no participant identifier appears in code; credentials come from the environment, and preflight checks only that a key is present, never reading or logging its value.

The synthetic stimuli in config/stimuli.example.json are invented sentences in arbitrary order. They are not a validated instrument and must not be used to measure anyone.

Rubric: Ortega, L., Iwashita, N., Norris, J. M., & Rabie, S. (2002, October). An investigation of elicited imitation tasks in crosslinguistic SLA research. Paper presented at the Second Language Research Forum, Toronto, Canada. Scoring criteria also draw on Ortega (2000).

License

MIT — see LICENSE. Copyright (c) 2026 Dovudxon Tojiyev.