Whisper timestamp support

[jcheng5]

Hello, I've been working on adding token level timestamp support for whisper models. It is working well locally but I have a few questions I'd like to pose before submitting a PR.

1. Claude Code is doing all the hard work. Some open source projects don't accept AI assisted code contributions, if that's the case here please let me know.
2. The approach in my branch is based on timing.py from openai/whisper (described in this notebook). It needs access to cross-attention weights from alignment heads. This unfortunately means that modifications to the scripts/whisper/export-onnx.py script were necessary, to add a new cross_attention_weights to the output, and existing published csukuangfj/sherpa-onnx-whisper-* models will not be able to support timestamps. Is this a big enough problem that you would decline the PR?
3. My branch currently outputs not only token-level timestamps, but also words and word-level timestamps (because timing.py supports this). Would the word-level output be worth keeping, or should I remove it so whisper output looks the same as other models? Update: I studied the word boundary logic and it is totally downstream of token timestamps, so I removed it--it is useful but should not be tied to any one model family.

Thank you for this great library!

[csukuangfj]

Claude Code is doing all the hard work. Some open source projects don't accept AI assisted code contributions, if that's the case here please let me know.

Are you able to fully understand the code you plan to contribute, and are you comfortable making changes during the code review process?

[jcheng5]

Are you able to fully understand the code you plan to contribute

Yes, although I'm not done doing the final review of all of it--I've reviewed the C++ code but there are some Python scripts relating to model export that I mostly vibe coded.

Also, I have to admit I don't deeply understand all of the math. For example, the links I provided in my previous message (timing.py, and that notebook) use a dynamic time warp algorithm that I don't understand, although I can see that "my" implementation is very close to the reference CPU implementation.

I'm totally comfortable making changes during code review.

The draft PR is here: #2945 and actually adds both token-level and segment-level timestamps.

I'm sorry, I ordinarily would not submit such a large PR to an open source project where I do not already have an established reputation. If it's too much, I can try to break it into smaller pieces. I discovered after posting the above message that I actually needed segment-level timestamps more than I needed token-level timestamps, and the segment-level timestamps are much simpler and don't require re-exporting the whisper models, just a fancy logit filter.

[jcheng5]

Since the PR exists now (#2945) I'll close this thread.