Skip to content

hotato 0.6.0

Choose a tag to compare

@quantumCF quantumCF released this 09 Jul 22:16

Added

  • hotato contract create/verify/inspect/pack/unpack, the portable failure
    contract
    : turns one real call moment into a self-contained <id>.hotato/
    bundle -- contract.json (schema hotato.contract.v1), the (clipped) audio,
    frame-level timing evidence, an input-health (trust) report, a shareable SVG
    card, a CI policy, and the exact replay/CI commands. create wraps the SAME
    round-trip scorability guarantee fixture create gives (via
    --from-candidate FILE#N, --stereo, --caller+--agent, or the opt-in
    --mono --diarize path): a not-scorable moment or a mono recording is
    refused with the honest reason (exit 2) and no bundle is written; the
    diarized-mono path never silently upgrades an indicative-only verdict, and
    honestly reports frame-level evidence as unavailable rather than fabricating
    it. verify re-scores a contracts directory (or one bundle) against each
    contract's own recorded policy and is the CI gate: exit 0 every contract
    passes, 1 a regression, 2 a usage error; emits text/JSON/HTML/JUnit.
    inspect prints one contract. pack/unpack round-trip a bundle through
    one deterministic single-file .hotato.pack archive with a sha256 manifest
    checked on unpack (a corrupted or tampered archive is refused, exit 2,
    nothing partial left behind). hotato card also renders a contract
    directly (kind voice-turn-taking-contract). Redacted by default (a
    candidate ref / source recording name is hidden unless
    --include-identifiers); no call ids, paths, or transcript text in any
    artifact. New module hotato.contract, new schema
    schema/contract.v1.json, docs docs/CONTRACTS.md.
  • hotato trace ingest/attach/export, the voice-trace observability
    bridge
    : ingest --otel FILE --out voice_trace.jsonl parses either a
    standard OTel JSON export (resourceSpans, best-effort span/event
    flattening) or hotato's own documented OTel bridge JSONL into schema
    hotato.voice_trace.v1 (caller/agent audio activity, TTS cancel/stop, ASR
    partials, tool calls, ...). attach BUNDLE --trace voice_trace.jsonl
    writes the trace into <bundle>/traces/voice_trace.jsonl and re-renders
    evidence/timeline.html with the trace's events drawn as a scale-aligned
    row, reading the bundle's OWN evidence/frames.jsonl and contract.json
    back in -- it never re-runs the VAD or diarizer, so it works on a
    diarized-mono bundle (no frame-level evidence) without the diarization
    extra installed, honestly noting the missing base timeline instead of
    fabricating one. export BUNDLE --format otel --out FILE writes the
    attached trace back out as the same bridge JSONL ingest reads, so
    ingest -> attach -> export -> ingest round-trips the identical spans.
    The evidence page states findings plainly -- "Evidence suggests TTS
    cancellation delay: cancel requested at 2.60s, audio stopped at 2.90s
    (delta 0.30s)." -- always followed by "Hotato does not prove root cause."
    and an explicit "Unknowns: no client-side playout trace was attached."
    line. Redacted by default (call id, agent id, ASR transcript text). New
    module hotato.trace, new schema schema/voice_trace.v1.json, docs
    docs/TRACE.md and docs/OTEL.md.
  • hotato run --mono call.wav --diarize, the opt-in, quality-gated
    mono-scorability front-end
    : a single-channel (mixed) recording -- until now
    the hard coverage wall, rejected as not scorable -- becomes scorable by running
    an off-the-shelf speaker diarizer over the mono to recover who was active when,
    reconstructing two caller/agent tracks, and feeding the EXISTING dual-channel
    scorer (zero engine edit). The two-channel path stays the gold reference; this
    widens coverage, honestly labeled. A pluggable diarizer-backend seam
    (mirroring the neural-VAD seam) ships three backends behind --diarizer:
    pyannote (local, offline, CPU-viable, default; [diarize]), sortformer
    (NVIDIA NeMo, local/GPU, best self-hostable on telephone;
    [diarize-sortformer]), and pyannoteai (HOSTED, best absolute, requires
    --egress-opt-in; [diarize-hosted]). A real per-file confidence gate
    (extending trust.py's scorability model) reads six signals -- speaker count,
    per-speaker activity, mean segmentation posterior, embedding cluster-separation
    margin, overlap ratio, segment churn -- into a separation_confidence and a
    tier: high scores normally (labeled diarized-mono), low scores but stamps
    indicative_only and fires NO SLA gate, refuse is not scorable (exit 2).
    Caller/agent assignment reuses the floor-dominance heuristic to PROPOSE a
    mapping stated as an assumption, overridable with --caller-speaker /
    --agent-speaker; a balanced mapping downgrades to indicative. On this path
    signals.echo / crosstalk is definitionally N/A (two slices of one microphone)
    and the echo gate never fires. hotato trust --stereo mono.wav --diarize
    reports the separability tier (high/low/refuse) WITHOUT scoring. No silent
    fallback anywhere: a missing extra / token / model raises a clean
    BackendUnavailable (exit 2) and NEVER scores raw mono. Honesty invariants:
    the default path (no --mono/--diarize) stays byte-identical (a mono file is
    still rejected as today), and a diarized-mono verdict is never presented as
    equivalent to a true dual-channel measurement. New module hotato.diarize
    (DiarizationResult, the backend registry, reconstruct_tracks,
    separation_confidence, assign_speakers, prepare_diarized_mono, the
    pyannote/sortformer/pyannoteai adapters, and a hermetic stub backend); new
    --mono/--diarize/--diarizer/--caller-speaker/--agent-speaker/
    --egress-opt-in flags on hotato run and --diarize/--diarizer on hotato trust; documented in docs/DIARIZE.md. The LIMITS.does_not_do /
    METHODOLOGY.md "no diarization" framing is superseded: two channels is the
    gold reference; mono is scorable via the opt-in, quality-gated front-end,
    labeled indicative below the bar. Speaker IDENTIFICATION is still out of scope
    (a diarizer assigns anonymous SPEAKER_00/01; it never says who a person is).
  • New optional extras [diarize], [diarize-sortformer], [diarize-hosted]
    (the [diarize] path raises the effective Python floor to >=3.10; the stdlib
    core stays >=3.9). Dependency licenses are logged in docs/DIARIZE.md and
    carried in the envelope's diarization.licenses block (pyannote-audio MIT;
    community-1 weights CC-BY-4.0; Sortformer streaming v2 CC-BY-4.0 -- the offline
    v1 is CC-BY-NC and is never shipped).
  • hotato card INPUT[#REF] --out card.svg, a shareable card from any hotato
    result
    : renders a DETERMINISTIC, stdlib-only, 1200x630 SVG with NO external
    resource (no font, image, stylesheet, script, or link; all color inline), so
    it drops straight into a PR, an issue, or a slide and looks the same forever.
    Four kinds are auto-detected: a talk-over candidate and a false-stop
    candidate (from a sweep/analyze candidate ref FILE#N), the
    threshold-funnel fix plan (decision: do_not_tune_single_threshold -- the
    hero card: "NO SINGLE THRESHOLD CAN FIX THIS", fix class: engagement-control),
    and a supported verify rollup ("FIX VERIFIED WITHOUT BREAKING
    BACKCHANNELS"). Every card names the MEASURED timing moment and never a verdict
    about intent, and carries no accuracy number. Redacted by default: a call
    id, a filesystem path (only a basename is ever shown), and a vendor recording
    name are hidden unless --include-identifiers. Exit 0 written / 2 bad input.
    New module hotato.card; three commit-ready assets under
    docs/assets/cards/ regenerated by scripts/render_card_assets.py; documented
    in docs/CARDS.md.
  • hotato start --demo, the guided, credential-less first run: one command,
    no account, no network. It sweeps the two bundled real demo calls, writes the
    sweep result (hotato-sweep.json), a self-contained HTML dashboard
    (hotato-sweep.html), and the threshold-funnel card
    (hotato-no-single-threshold.svg), then turns one real missed-interruption
    candidate into a demo failure contract (contracts/demo-missed-interruption.hotato,
    --expect yield) and verifies it immediately -- it genuinely fails, so the
    loop is visible end to end in one command: a real failure becomes a
    candidate, becomes a portable contract, and contract verify catches it.
    It then prints the exact next commands: promote a candidate into a
    permanent fixture, run those fixtures in CI, re-verify the demo contract,
    and render a card. The --stack/--folder/--stereo modes are
    placeholders in this build and route to hotato sweep/analyze/run.
    Exit 0 done / 2 usage. New module hotato.start; documented in
    docs/START.md.

Fixed

  • hotato.diarize pyannote 4.x compatibility: Pipeline.from_pretrained
    now tries the token= kwarg first and falls back to the removed
    use_auth_token= name on TypeError, and a new _unpack_pipeline_output
    branches on the DiarizeOutput object pyannote.audio >=4.0 returns from a
    pipeline call (previously unpacked as a 3.x (Annotation, embeddings)
    tuple, raising TypeError/AttributeError against a 4.x install) so both
    pyannote 3.x and 4.x load and score cleanly. _embedding_margin no longer
    divides by a zero norm and fabricates cos = 0 (read by the confidence
    gate as adequate separation) for a degenerate (zero-norm / non-finite)
    speaker centroid; it now returns None -- "no margin available" -- the
    same honest no-signal result a missing embeddings array already gave.
  • New yield-boundary confidence gate signal (signals.yield_boundary):
    benchmarked against a real pyannote community-1 backend over the AMI
    corpus (in-repo harness: tools/bench_diarize/, dev-only, never shipped),
    the existing six diarization-quality signals measured clean, well-separated,
    stable speakers but were anti-correlated with verdict correctness -- the
    high confidence tier reproduced the dual-channel did_yield verdict
    LESS often than low, concentrated in short-yield, backchannel, and
    sub-second talk-over cases, and present even at DER 0.000, because the
    verdict turns on a sub-250 ms agent-quiet gap that DER's collar and the
    quality signals forgive. The new 7th signal replays the engine's yield
    logic directly over the diarization timelines (no model calls, no
    reconstruction), perturbs the speaker boundaries by +/- 0.25s, and checks
    the verdict survives; a yield resting on a briefer-than-0.5s caller run
    (backchannel-grade) or that flips under the boundary nudge is barred from
    the high tier (drops to low, indicative-only) even when the other six
    signals look clean. Honest high coverage on real material shrinks as a
    result -- that is the point: high now requires a boundary-robust
    verdict, not just clean diarization. The embedding-margin signal itself
    measured uninformative on the AMI benchmark (clustered ~0.43-0.52
    regardless of verdict correctness); left as measured, redesign deferred to
    a future gate-recalibration stage rather than tuned blind here.