word_timestamps: punctuation inside a word starts a new "word" (3.5% -> 3, .5, %; "안녕"이라고 -> "안녕, "이라고)

[sungbin1015]

With word_timestamps=True, an ASCII punctuation character that sits inside a space-delimited word splits that word, and the punctuation ends up at the start of the second half. Decimals, thousands separators, times and domain names are affected in every language that goes through split_tokens_on_spaces. Korean is hit harder, because particles attach directly after a closing quote or bracket ("안녕"이라고, '서울'에서, (주)한국으로), so the closing quote is moved onto the particle.

Checked on main at 8609812 (Python 3.11, torch 2.14.1+cpu, tiktoken 0.14.0).

1. Real audio, tiny, CPU

Audio: EN-UK inflation rate increases to 4%.ogg (Wikinews audio, CC BY 2.5, 3 min; only the first 60 s are used).

import sys
import whisper

model = whisper.load_model("tiny", device="cpu")
result = model.transcribe(
    sys.argv[1], language="en", temperature=0.0, fp16=False,
    word_timestamps=True, clip_timestamps="0,60",
)
for segment in result["segments"]:
    words = segment["words"]
    for i, w in enumerate(words):
        if i > 0 and not w["word"].startswith(" "):
            pair = [(x["word"], float(x["start"]), float(x["end"])) for x in words[i - 1 : i + 1]]
            print(f"{segment['text']!r}\n    {pair}")

Output:

' This is a spoken wiki news article from wikinews.org, the free news source, read by Kevin'
    [(' wikinews', 2.96, 3.6), ('.org,', 3.6, 4.16)]
' of the United Kingdom measured by the consumer price index measured to 4% in January 2011.'
    [(' 4', 23.84, 24.14), ('%', 24.14, 24.58)]
' In December 2010, this figure was 3.7%.'
    [(' 3', 29.48, 29.78), ('.7%.', 29.78, 31.3)]
' According to BBC News Online, 4% is the highest CPI that the UK has experienced since'
    [(' 4', 33.78, 34.0), ('%', 34.0, 34.38)]
' increasing from 17.5% to 20% on January 4%, and the value of the crude oil rising.'
    [(' 17', 45.42, 45.82), ('.5', 45.82, 46.5)]
' increasing from 17.5% to 20% on January 4%, and the value of the crude oil rising.'
    [('.5', 45.82, 46.5), ('%', 46.5, 46.9)]
' increasing from 17.5% to 20% on January 4%, and the value of the crude oil rising.'
    [(' 20', 47.06, 47.38), ('%', 47.38, 47.8)]
' The inflation rate of the retail price index is the country has increased to 5.1%, previously'
    [(' 5', 55.84, 56.08), ('.1%,', 56.08, 57.36)]
' the value was 4.8%.'
    [(' 4', 58.62, 58.86), ('.8%.', 58.86, 59.82)]

17.5% is three words ( 17, .5, %), 3.7%. is two ( 3, .7%.), and wikinews.org, is wikinews + .org,.

2. No model: tokenizer + merge_punctuations + WriteSRT

The split depends only on the text tokens, so it can be shown without audio. The Korean lines below are the tokenizer applied to text I typed, not model output.

import io

from whisper.timing import WordTiming, merge_punctuations
from whisper.tokenizer import get_tokenizer
from whisper.utils import WriteSRT

PREPEND = "\"'“¿([{-"  # transcribe() defaults
APPEND = "\"'.。,，!！?？:：”)]}、"


def words_for(text, language):
    tok = get_tokenizer(True, num_languages=100, language=language, task="transcribe")
    tokens = tok.encode(text)
    words, word_tokens = tok.split_to_word_tokens(tokens)
    alignment = [WordTiming(w, t, 0, 0, 1) for w, t in zip(words, word_tokens)]
    merge_punctuations(alignment, PREPEND, APPEND)
    return [a.word for a in alignment if a.word]


for language, text in [
    ("ko", ' 그는 "안녕"이라고 말했다.'),
    ("ko", " 그는 “안녕”이라고 말했다."),
    ("ko", " '서울'에서 (주)한국으로 갔다."),
    ("ko", " 물가가 3.5% 올랐고 1,234명이 10:30에 왔다."),
    ("en", " It rose 3.5% to $1,234.50 at 10:30 on example.com."),
]:
    print(repr(text))
    print("   ", words_for(text, language))

# what the subtitle writer does with such a word list
words = words_for(' 그는 "안녕"이라고 말했다.', "ko")
segment = {
    "start": 0.0,
    "end": float(len(words)),
    "text": "".join(words),
    "words": [
        {"word": w, "start": float(i), "end": float(i + 1)} for i, w in enumerate(words)
    ],
}
out = io.StringIO()
WriteSRT(".").write_result({"segments": [segment]}, out, max_words_per_line=2)
print(out.getvalue())

Output:

' 그는 "안녕"이라고 말했다.'
    [' 그는', ' "안녕', '"이라고', ' 말했다.']
' 그는 “안녕”이라고 말했다.'
    [' 그는', ' “안녕”이라고', ' 말했다.']
" '서울'에서 (주)한국으로 갔다."
    [" '서울", "'에서", ' (주', ')한국으로', ' 갔다.']
' 물가가 3.5% 올랐고 1,234명이 10:30에 왔다.'
    [' 물가가', ' 3', '.5', '%', ' 올랐고', ' 1', ',234명이', ' 10', ':30에', ' 왔다.']
' It rose 3.5% to $1,234.50 at 10:30 on example.com.'
    [' It', ' rose', ' 3', '.5', '%', ' to', ' $1', ',234', '.50', ' at', ' 10', ':30', ' on', ' example', '.com.']
1
00:00:00,000 --> 00:00:02,000
그는 "안녕

2
00:00:02,000 --> 00:00:04,000
"이라고 말했다.

The same sentence with a curly closing quote (”) gives one word “안녕”이라고; with an ASCII " it gives "안녕 and "이라고, and the second subtitle starts with the closing quote.

Expected

A word is what stands between two spaces: 3.5%, $1,234.50, 10:30, example.com., "안녕"이라고. Or, if punctuation is meant to separate words, at least not .5 / "이라고 with the separator glued to the front of what follows.

Where it comes from

Tokenizer.split_tokens_on_spaces, whisper/tokenizer.py lines 317-325:

special = subword_tokens[0] >= self.eot
with_space = subword.startswith(" ")
punctuation = subword.strip() in string.punctuation
if special or with_space or punctuation or len(words) == 0:
    words.append(subword)
    word_tokens.append(subword_tokens)
else:
    words[-1] = words[-1] + subword
    word_tokens[-1].extend(subword_tokens)

1. A punctuation-only subword always opens a new word, even with no space before it (line 319-320): 3 | ..
2. The next subword has no leading space, so it is appended to words[-1], which is now the punctuation (line 324): . + 5 = .5.
3. merge_punctuations (whisper/timing.py line 268) only moves a word that is entirely in append_punctuations; .5 and "이라고 are not, so they stay. Setting prepend_punctuations=""/append_punctuations="" does not help, because the gluing happens in step 2.

string.punctuation is ASCII only and the test is a substring test, which explains the inconsistencies above: ” is not in it, so the curly quote does not split; the single token %. is not a substring of it, so 4%. stays one word while 4 + % becomes two.

Question / proposal

Is the .5 / "이라고 form intended? If not, would you accept rejoining a punctuation "word" with the word before it when the next subword has no leading space? Locally this is 10 lines in the else: branch:

else:
    while (
        len(words) > 1
        and words[-1]
        and words[-1] in string.punctuation
        and word_tokens[-2][0] < self.eot
    ):
        subword = words.pop() + subword
        subword_tokens = word_tokens.pop() + subword_tokens
    words[-1] = words[-1] + subword
    word_tokens[-1].extend(subword_tokens)

With it the script in section 2 prints

[' 그는', ' "안녕"이라고', ' 말했다.']
[" '서울'에서", ' (주)한국으로', ' 갔다.']
[' 물가가', ' 3.5', '%', ' 올랐고', ' 1,234명이', ' 10:30에', ' 왔다.']
[' It', ' rose', ' 3.5', '%', ' to', ' $1,234.50', ' at', ' 10:30', ' on', ' example.com.']

and the existing test suite still passes (pytest -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' -m 'not requires_cuda': 20 passed). Punctuation at the end of a word is untouched, so a trailing % is still its own word unless it is added to append_punctuations; I left that alone because it is a separate decision. Token order and the token-to-word mapping are unchanged, only the grouping differs. I can open a PR with the change and a test if that is the direction you prefer.

Related work

I did not find an earlier report of this. Closest items: #869 (introduced word timestamps and split_tokens_on_spaces; the review discussed attaching punctuation to adjacent words, not punctuation inside a word), #973 (exposed prepend_punctuations/append_punctuations), #1089 (replacement-character handling in split_tokens_on_unicode, different function and input), discussion #1243 (suggests empty prepend_punctuations/append_punctuations to get punctuation as separate words; that does not change the result here), open PRs #2863, #2847 and #2793 (subtitle writer in whisper/utils.py, not the word split). Searched discussions, PRs and issues for: split_tokens_on_spaces, split_to_word_tokens, string.punctuation, merge_punctuations, prepend_punctuations, append_punctuations, "word timestamps" combined with decimal / number / comma / percent / dollar / quote / apostrophe / parenthesis / hyphen / URL / punctuation / Korean.

[MohamedMady19]

Nice, precise report. As far as I can see, the split comes from split_tokens_on_spaces in whisper/tokenizer.py: a subword starts a new word not only when it begins with a space, but also when subword.strip() in string.punctuation. So a token that is just ., ,, % or " opens a new word even in the middle of a space-delimited word, and the following tokens are glued onto it (.5, "이라고). merge_punctuations in timing.py only re-attaches punctuation that stands alone as its own word, so it cannot repair .5 or "이라고.

(A side note: x in string.punctuation is a substring test, not a set membership test, so "" and runs like ",-." also count as punctuation.)

Workaround without patching Whisper. Your own detection rule (a word that does not start with a space continues the previous one) works as a post-processing step:

def merge_inword_punctuation(segments):
    """Re-join pieces of one space-delimited word that word_timestamps split at ASCII punctuation."""
    for seg in segments:
        merged = []
        for w in seg.get("words", []):
            if merged and not w["word"].startswith(" "):
                prev = merged[-1]
                prev["word"] += w["word"]
                prev["end"] = w["end"]
                prev["probability"] = min(prev["probability"], w["probability"])
            else:
                merged.append(dict(w))
        seg["words"] = merged
    return segments

result["segments"] = merge_inword_punctuation(result["segments"])

On your examples this gives ' 3.5%' (start of 3, end of %) and ' "안녕"이라고' as single words. It only applies to languages that go through split_tokens_on_spaces; Chinese, Japanese, Thai, Lao, Burmese and Cantonese use split_tokens_on_unicode, where every piece is meant to be its own unit.

Possible fix upstream: start a new word on punctuation only when the punctuation token itself begins with a space, i.e. drop the bare punctuation condition and rely on with_space. Sentence-final punctuation then stays attached to its word, which merge_punctuations would have done anyway. It would need a run of the existing timing tests, since it changes which tokens form a word.