language	ja
license	apache-2.0
library_name	downstep
tags	text-to-speech japanese pitch-accent prosody accent-phrase openjtalk voicevox calibration selective-prediction onnx
datasets	unidic naist-jdic
metrics	exact_match f1

Downstep — pitch accent for the compounds your dictionary has never seen

Getting 東京大学 right is a lookup. The hard part is 新型感染症対策本部.

Japanese TTS front ends get accent from a dictionary. When the word is in the dictionary they are excellent. When it is not — and new compounds are minted constantly — they fall back to composing the accent from the parts, and that is where they break.

Downstep is a 35M-parameter model for that one gap. It predicts the accent nucleus and the accent-phrase boundaries of a compound from its morphemes, it reports a calibrated confidence, and below a threshold it abstains and hands the word back to the existing system unchanged.

The hypothesis, stated so it can be refuted:

On dictionary-listed compounds Downstep should not beat a rule-based front end and does not need to. On compounds the dictionary lacks, the ordering should reverse and the gap should be large.

Model: https://huggingface.co/NagaYu/downstep (PyTorch, ONNX fp32, ONNX int8, plus the vocab and calibration the checkpoint needs.) Benchmark: https://huggingface.co/datasets/NagaYu/downstep-bench (89,539 compounds) Demo: https://huggingface.co/spaces/NagaYu/downstep — hear both accents on the same voice

It does. On the same 5,414 test compounds, when the dictionary entry is present OpenJTalk scores 96.4% and Downstep 81.3%; when the entry is removed OpenJTalk drops to 40.6% and Downstep — which never had the entry — stays at 81.3%.

That is the whole claim, and it comes with a cost that is reported here rather than buried: adopting Downstep at full coverage makes 256 of those 5,414 words worse while making 2,462 better. Guaranteeing that no individual word regresses is possible, but only by abstaining on 93% of them. See The honest cost.

---

Results

Same 5,414 test compounds in both conditions — those whose reading OpenJTalk produced correctly with and without its dictionary entry, so the comparison is about accent and not about G2P. Gold accents come from NAIST-jdic and UniDic. Accent nucleus exact match (%).

The compound is in the dictionary

system	all	common nouns	proper nouns	hard
(A) OpenJTalk	96.4	95.7	96.8	96.9
(B) dictionary lookup only	100.0	100.0	100.0	100.0
(C) Downstep, no abstention	81.3	79.7	82.3	73.3
(D) Downstep + abstention	96.3	—	—	—

(B) is 100% by construction: the gold is the dictionary entry. That is the point of this half of the table — the listed split cannot show that a system is good, only that it does no damage. Downstep alone is clearly worse here, and the right behaviour is to defer, which is what (D) does and what DropInAdapter does by default.

The compound is not in the dictionary

The entry is ablated per-target so the front end must compose the accent from the parts, while the gold stays real. Same words, same gold.

system	all	common nouns	proper nouns	hard	hard (no component seen)
(A) OpenJTalk	40.6	31.4	46.1	40.9	31.6
(B) dictionary → falls back to (A)	40.6	31.4	46.1	40.9	31.6
(C) Downstep, no abstention	81.3	79.7	82.3	73.3	62.2
(D) Downstep + abstention	81.3	—	—	—	—

+40.7 points overall, and +32.4 on the hard split — compounds whose final morpheme never appeared in any training compound. That last column matters: a lookup-shaped model gains nothing there (we measured 58.3% → 58.3% in a pilot), so the gain is not memorisation.

OpenJTalk also wrongly splits the compound into two accent phrases 23.3% of the time once it loses the entry; Downstep does so 2.2% of the time.

By word origin (語種), unlisted condition: 和 36.3→79.1, 漢 42.0→84.1, 外 55.1→82.2, 混 24.4→87.8, 固 41.0→81.5. By component count: 2 parts 41.7→81.9, 3 parts 36.3→78.5, 4 parts 21.2→77.5 — the baseline degrades sharply as compounds get longer, and Downstep does not.

The honest cost

"Never worse" is the claim that matters for adoption, and it does not come free.

operating point (threshold chosen on dev, reported on test)	coverage	accuracy	words made worse	words made better
unlisted, max accuracy	100%	81.3%	256	2,462
unlisted, zero-regression on dev	7.3%	42.2%	0	88
listed, max accuracy	15.8%	96.3%	27	24
listed, zero-regression on dev	1.4%	96.4%	1	3

Read the first two rows together. In aggregate the combined system is at or above OpenJTalk at every coverage level — that is the shaded region below. But per word, answering everything trades 256 regressions for 2,462 improvements (about 10:1), and the only way to drive individual regressions to zero is to abstain on almost everything, which throws away most of the benefit. Which of those you want is a product decision, not a technical one, so both are shipped and neither is hidden.

Calibration, n-best, speed

• Calibration. Temperature scaling fitted on dev: ECE 0.048 → 0.011 (T = 1.498).
• n-best. Top-1 81.3%, top-2 94.4%, top-3 98.0%, top-5 99.7%. On the 49 test words whose dictionaries attest more than one accent, top-1 is 61.2% but top-3 is 98.0% — forcing a single answer measures the wrong thing when the language genuinely varies.
• Speed, CPU (4 threads): Downstep 2.0 ms/word; OpenJTalk 0.45 ms/word. Downstep is slower, and it is only invoked for words the dictionary lacks.

---

How it works

flowchart LR
    A["input: 新型感染症対策本部"] --> B{"in the<br/>dictionary?"}
    B -- "yes" --> C["dictionary lookup<br/>(unchanged behaviour)"]
    B -- "no" --> D["MoraTokenizer<br/>シ・ン・ガ・タ・カ・ン・…"]
    D --> E["morpheme analysis<br/>新型+感染+症+対策+本部<br/>word origin, part accents, sandhi class"]
    E --> F["JointPredictor 35M<br/>nucleus + phrase boundary, jointly"]
    F --> G["constrained decode<br/>at most one nucleus per accent phrase"]
    G --> H{"calibrated<br/>confidence ≥ threshold?"}
    H -- "yes" --> I["Downstep's answer"]
    H -- "no (abstain)" --> J["return OpenJTalk's<br/>answer unchanged"]
    C --> K["DropInAdapter<br/>NJD / VOICEVOX / HTS labels"]
    I --> K
    J --> K
    K --> L["VOICEVOX · Style-Bert-VITS2"]

Loading

Terms used below

Japanese accent terminology is kept because the field's terms are precise and the code uses them; each is glossed here once.

term	meaning
accent nucleus (アクセント核)	the mora after which the pitch falls. A word has at most one per accent phrase
平板 heiban	no nucleus — pitch does not fall. Written accent = 0
頭高 atamadaka	nucleus on mora 1
中高 nakadaka	nucleus somewhere in the middle
尾高 odaka	nucleus on the last mora — audible only in what follows, which is why it is easily confused with 平板
語種 goshu	word origin: 和 native, 漢 Sino-Japanese, 外 loanword, 混 mixed, 固 proper name
連濁 rendaku	voicing of a non-initial element's first consonant in a compound (山 yama → 神山 kamiyama)
促音化	a stem-final consonant becoming the geminate ッ in compounding
拗音 / 促音 / 撥音 / 長音	palatalised moras (キャ), the geminate ッ, the moraic nasal ン, and the long-vowel mark ー

The four components:

1. MoraTokenizer — kana → moras, with 拗音 fused onto the preceding kana and 促音 ッ, 撥音 ン and 長音 ー each their own mora. Validated against the mora counts two dictionaries record independently: 99.982% agreement over 788,331 entries.
2. JointPredictor — 35,054,595 parameters. Predicts nucleus and accent-phrase boundary jointly from per-mora features that make morpheme boundaries, the final morpheme, 語種 and each component's own dictionary accent explicit.
3. CalibratedAbstain — temperature-scaled confidence and a threshold. Below it, the fallback's Prediction object is returned itself, not a lookalike.
4. DropInAdapter — returns accent phrases in OpenJTalk/VOICEVOX shape.

---

Install and use

pip install -e ".[all]"

downstep accent "新型感染症対策本部" --nbest 3

mora      シ ン ガ タ カ ン セ ン ショ ー タ イ サ ク ホ ン ブ
pitch     ▁┌ ▔▔ ▔▔ ▔▔ ▔▔ ▔▔ ▔▔ ▔▔ ▔▔▔▔ ▔▔ ▔▔ ▔▔ ▔▔ ▔▔ ▔┐ ▁▁ ▁▁
accent    15 (中高) -- fall after mora 15 of 17

Other subcommands: downstep compare (side by side with OpenJTalk), downstep lexicon (every attested accent for a word, including variation), downstep bench (latency).

Dropping it into VOICEVOX / Style-Bert-VITS2

Both stacks get accent from pyopenjtalk. The adapter returns the same shapes, so the change is where the accent comes from, not what the rest of the pipeline does.

from downstep.adapters.openjtalk import DropInAdapter

adapter = DropInAdapter.load("checkpoints/base")   # dictionary first, model for the rest

# 1. VOICEVOX-shaped accent phrases: [{"moras": [...], "accent": int, "pause_mora": None}]
phrases = adapter.accent_phrases("新型感染症対策本部を設置した")

# 2. NJD features, same keys as pyopenjtalk.run_frontend() -- swap it in directly
njd = adapter.njd_features("新型感染症対策本部を設置した")

# 3. HTS full-context labels, for anything that synthesises from labels
labels = adapter.full_context_labels("新型感染症対策本部を設置した")

For VOICEVOX, replace the accent phrases returned by its AudioQuery construction with adapter.accent_phrases(text) and leave mora timing and pitch to the engine. For Style-Bert-VITS2, its g2p path calls pyopenjtalk.run_frontend; point that at adapter.njd_features instead.

One trap, and it is the reason to use the adapter rather than rolling your own. In HTS full-context labels the accent field F2 writes 平板 as F2 == F1, not as 0 — so F2 == F1 is ambiguous between 平板 and 尾高, and writing a literal 0 there silently corrupts every flat-accent word. Reading F2 naively scores OpenJTalk at 70% where the truth is 94%. The adapter converts in both directions and refuses the ambiguous case rather than guessing.

Try it

app.py is a Gradio Space: type a sentence, see Downstep's and OpenJTalk's pitch contours on the same axes with the differing moras marked, and hear both — the two versions are synthesised with the same voice so the only difference is the accent.

python app.py

A pre-rendered version of the same comparison — seven compounds, both accents, both audible — is at https://huggingface.co/spaces/NagaYu/downstep, built by scripts/build_static_space.py. It is a static page because Gradio Spaces are not available on this account's tier; typing your own word needs the local app above.

Exports

python scripts/export.py --checkpoint checkpoints/base --out artifacts/export

artifact	size	vs torch	numeric check
ONNX fp32 (opset 17)	134.3 MiB	3.4 ms vs 17.0 ms per word (1×4)	max abs diff 2.1e-05 — pass
ONNX int8 (dynamic)	34.4 MiB (3.91× smaller)	2.7–3.0× faster than fp32	max abs diff 1.3e-01 — fails a 5e-02 tolerance

The int8 model's decisions are identical — nucleus argmax and boundary sign agree with fp32 on 100% of checked inputs — but its logits move enough to fail the tolerance. That matters for the calibrated confidence, which is what the abstention gate thresholds, so the int8 build should be re-calibrated before it is used with a threshold rather than inheriting the fp32 one. The export script reports this as a failure instead of widening the tolerance until it passes.

There is no GGUF. scripts/export.py explains why rather than shipping a file that looks like a model: llama.cpp dispatches inference on general.architecture and has no downstep entry, and its interface (token ids in, vocabulary logits out) does not describe this model, which takes 13 parallel feature streams and returns three per-mora logit vectors. Producing a runnable GGUF is a C++ change to llama.cpp, not something an export script can do. ONNX is the export that actually runs.

The --push flag exists, defaults to off, and refuses to upload on its own; it prints the exact manifest it would send. Publishing is a maintainer decision.

---

Reproducing

make bench      # build the benchmark from the two dictionaries   (~1 min)
make train      # train the base model                            (~82 min, Apple MPS)
make eval       # four systems x two conditions x strata          (~2 min)
make figures    # redraw everything in figures/ from results.json
make test

Every number in this README comes from benchmarks/results.json, which comes from scripts/evaluate.py. Nothing is transcribed by hand.

---

The data, and what is not in it

Two independent, redistributable accent lexicons, both read offline straight from their compiled MeCab dictionaries:

source	what it gives	licence
UniDic 2.1.2 (NINJAL / UniDic Consortium)	accent, 語種, accent-sandhi class, and 21.5k nouns with more than one attested accent	GPL / LGPL / BSD-3 (the BSD option is used)
NAIST-jdic (NAIST, via pyopenjtalk)	456k accented nouns including lexicalised compounds; it is also OpenJTalk's own dictionary	BSD-3

The benchmark is 89,539 compounds (71,544 train / 8,959 dev / 9,036 test), each decomposing into ≥2 morphemes whose readings compose to the compound's reading under 連濁 and 促音化. Splits are disjoint at the compound level. The hard subset (818 test items) is compounds whose final morpheme never occurs in any training compound. 1,030 items carry more than one attested accent and are scored with all of them accepted.

Decomposition comes from fugashi + UniDic, deliberately a different dictionary from the NAIST gold, so the parts handed to the model are not derived from the answer.

What is not here

• No human annotation. Every label is a dictionary entry. The compounds Downstep exists for are exactly the ones no dictionary has, and their accent can only be settled by asking native speakers. What ships instead is the whole apparatus for doing that: docs/ANNOTATION_GUIDELINES.md, the multi-annotator CSV format, the agreement statistics, and scripts/extract_compounds.py, which mines genuinely novel compounds from text and emits them unlabeled. No number here should be read as if a person had checked it.
• The "unlisted" condition is simulated, not observed. It is a held-out-entry ablation on lexicalised compounds, which is what makes real gold available at all. Ablation is done one compound at a time: batching corrupts it badly (only 52.3% of targets survive unchanged — 一人娘 becomes イチニンムスメ instead of ヒトリムスメ).
• Boundary F1 is not a hard task here. Dev boundary F1 reaches 1.000, but the only positive boundary examples available offline are constructed (two compounds joined by a case particle), and that is easy. The meaningful boundary result is the negative one in the table above: how often a system wrongly splits a compound.
• Proper nouns are 63% of the benchmark. Place and person names are a different, more arbitrary problem, so they are always reported as their own stratum, never averaged in silently.
• This is not a speech-corpus method. Accent models trained on thousands of hours of audio are solving a different problem with different data. Downstep reads text and a dictionary, runs on a CPU, and claims one narrow thing: dictionary-unlisted compounds.

On variation

Standard (Tokyo) accent is the default here because that is what these two lexicons encode — a statement about available data, not about how anyone should speak. Regional and generational accent patterns are ordinary variation, not error. The evaluation is built to respect that: where the dictionaries attest several accents for a word, all are counted correct, and n-best exists for the same reason. Non-Tokyo systems belong in downstep/adapters/dialect.py as their own adapters; that interface ships, with no invented accent data behind it, because no licensed non-Tokyo accent resource was available offline.

---

Attribution

Contains data derived from UniDic 2.1.2, © The UniDic Consortium, used under the BSD 3-clause option of its tri-licence, and from the NAIST Japanese Dictionary, © Nara Institute of Science and Technology, BSD 3-clause. Full notices in NOTICE. Code is Apache-2.0. Derived benchmark rows (surface, reading, accent) are redistributed; the dictionaries themselves are not.

See docs/METHODOLOGY.md for the experimental design and its threats to validity, and cards/DATASET_CARD.md for the dataset card.