Skip to content

v0.13.2

Choose a tag to compare

@xr843 xr843 released this 19 Oct 12:09
bbe7eab

Fewer misquotations from three masters. master-nagarjuna, master-tsongkhapa and
master-fazang now carry six more CBETA-verified passages that Claude kept quoting from
memory — several of them wrongly — and master-tsongkhapa states his reversal of the common
formulation (appearance refutes existence, emptiness nonexistence) instead of letting the
model substitute it. The evaluation's quotation check now flags 500 unverified quotations
across the 240-fixture Claude run instead of 1,838: the rest were glossed terms, citation
lines and parenthetical notes, and every misquotation found by hand is still flagged.

  • Six passages claude-sonnet-4-6 kept quoting from memory, often wrongly, are now in the
    personas' checked excerpts, each verified clause by clause against its CBETA fascicle:
    master-nagarjuna gets 中论 卷1 「已去无有去……」 and 十住毗婆沙论 卷1 on the first
    ground; master-tsongkhapa gets the 法尊 rendering of 中论 24.18 as 广论 卷17 quotes it,
    and 广论 卷1 with the line it quotes from 《般若灯广释》 marked as a quotation;
    master-fazang gets 五教章 卷4 因陀罗网境界门 and 「初发心时,便成正觉」 from the 60-fascicle
    华严经 (now declared, T09n0278), which the model had been citing as the 80-fascicle one.
    master-tsongkhapa's teaching notes now state his reversal — appearance refutes the extreme
    of existence, emptiness the extreme of nonexistence — which the model had been replacing
    with the common formulation.

Fixed — unverified_quotes counts quotations, not every pair of quotation marks (2026-10-19)

The v1.0 gate asks for a ruling on every unverified_quotes entry, and the
claude-sonnet-4-6 sweep (r2, 240 answers) had 1838 of them. A CBETA hand check of
the tsongkhapa / fazang / nagarjuna misses found 52 of 100 were not quotations at all
and 21 were already in the local excerpts. scripts/_citation_evidence.py now:

  • Extracts only spans presented as a source's words: a citation inside or right
    after the span, a speech verb or title right before it (「经云:」「明言」「》」), a
    colon or dash introducing it, traditional characters copied from a source, a span
    standing alone on its line, a blockquote wrapped in quotation marks, or a citation
    later in the same paragraph. Spans under 8 Chinese characters need a citation beside
    them, a speech verb right before them, or a whole sentence after a colon — otherwise
    they are terms (「无我」「执筏」). Callouts, disclaimers, misconceptions, suggested
    follow-up questions and citation-only blockquotes are no longer reported.
  • Matches what the model actually quoted: citation lines, links and 「出处:」 labels
    inside a blockquote, and inserted parenthetical glosses (「我说即是无(空)」) are not
    part of the quoted words. A quotation split by 「……」「…」「(中略)」 matches only
    if every segment is verbatim in one and the same declared work — two genuine halves
    from two books stay unverified. There is still no fuzzy matching: 「无」 written as
    「空」 stays unverified.
  • Reports a blockquote that is one quotation once, not once as the blockquote and
    again as the quotation inside it.

Re-graded offline over the archived r2 answers: 1838 → 500 (1835 → 500 counting
from the old rules on this tree, whose new excerpts already settle three); ajahn-chah 116→43,
atisha 50→17, buddhaghosa 119→34, compare-masters 299→84, fazang 59→16, huineng 70→25,
kumarajiva 66→11, mahasi-sayadaw 122→31, master-curriculum 120→21, master-debate
166→37, milarepa 77→21, nagarjuna 127→20, ouyi 72→20, tsongkhapa 120→31, xuanzang
45→5, xuyun 108→49, yinguang 67→26, zhiyi 35→9. Every r2 quotation the hand check
judged near, other_work, not_found or paraphrase is still reported; a random 30 of the
dropped entries were each confirmed by hand to be a non-quotation or verbatim in the
local excerpts. The grader fingerprint changes, so earlier reports re-grade.