Possible Paraformer timestamp issues in the Hugging Face full pipeline and GPU dynamic batching

[John6666cat]

I originally picked this up from the Hugging Face Forum thread here:

FunASR using Hugging Face Hub with paraformer-zh errors

The text below was AI-generated from my notes and test results. I ran and checked the reproductions, comparisons, and linked notebooks myself.

---

I may be misunderstanding some intended API semantics, but I can reproducibly separate what looks like two timestamp-related issues in the current FunASR Paraformer path.

Since new Issues currently appear to be restricted in this repository, I am posting the reproduction here first. If these should become separate Issues or PRs, I can split them.

The setup itself seems intended to be supported: the funasr/paraformer-zh Hugging Face model card documents a full pipeline with VAD, punctuation, and speaker diarization, with timestamps and speaker labels in the output.

Environment

The real-model reproduction was run with:

FunASR: 1.4.3
PyTorch: 2.11.0+cu128
GPU: NVIDIA T4

ASR:
  Hugging Face funasr/paraformer-zh

Runtime predictor:
  CifPredictorV2
  tail_mask=True
  tail_threshold=0.45

The full environment and executed outputs are available in the public notebook linked below.

---

1. Possible timestamp-request propagation mismatch

With the Hugging Face Paraformer full pipeline, timestamps were missing unless I explicitly added:

pred_timestamp=True

For example:

result = model.generate(
    input="meeting.wav",
    pred_timestamp=True,
)

I reproduced the timestamp recovery on both CPU and T4.

What makes this look like an internal integration mismatch is that current AutoModel requests timestamps when a speaker model is present using:

cfg["output_timestamp"] = True
cfg["return_time_stamps"] = True

while plain Paraformer.inference() uses:

pred_timestamp = kwargs.get("pred_timestamp", False)

for its timestamp-generation branch.

So the effective path appears to be roughly:

speaker model enabled
        |
        v
AutoModel requests timestamps
  output_timestamp=True
  return_time_stamps=True
        |
        v
Paraformer
        |
        +-- checks pred_timestamp
        |
        +-- pred_timestamp remains False
                |
                v
          no ASR timestamp

The runtime A/B was:

same HF Paraformer model

without pred_timestamp=True
    -> timestamp missing

with pred_timestamp=True
    -> timestamp present

If pred_timestamp is intentionally a separate user-facing requirement for Paraformer, then perhaps this is mainly a documentation/API-contract issue.

If the full speaker pipeline is expected to request the timestamp automatically, then this looks more like a flag-propagation mismatch.

There is an older related report involving missing Paraformer timestamps with VAD:

Issue #1747: paraformer-en + VAD result has no timestamp

I do not assume that issue has exactly the same root cause, but it seems related enough to mention.

Secondary schema symptom

When the timestamp path was missing, I also observed a different sentence_info fallback schema.

The fallback record contained:

"sentence"

while the normal timestamp-enabled path contained the documented:

"text"

This can turn the earlier timestamp failure into a later:

KeyError: 'text'

when application code follows the model-card example:

sentence["text"]

So I would treat that KeyError as a secondary symptom rather than the first failure.

---

2. Separate GPU dynamic-batch timestamp-padding issue

After explicitly enabling pred_timestamp=True, I found a second issue.

This one is specifically associated with unequal-length VAD segments grouped into an ordinary GPU dynamic batch.

I used an exact public 20.000 s AISHELL-4 crop.

Input identity:

Dataset:
  ggfox00000/dia-aishell4-test

Revision:
  f2415e0e2749c94a2b48fa9b24d94b406a26ea00

Source file:
  audio/test/M_R003S01C01.flac

Crop:
  1315.1222 s -> 1335.1222 s

Duration:
  20.000 s

Crop SHA-256:
  7b3dad101a169506bc274fefa7fb7db278e941f24e780d2a55c50d23c4c23937

Observed result:

Condition	Max timestamp end	Backward timestamp jumps
Physical input	20.000 s	—
Unmodified GPU dynamic batch	24.710 s	1
Patched GPU dynamic batch	20.030 s	0
Patched forced-single reference	20.030 s	0

The abnormal timestamp tail therefore was:

24.710 - 20.000 = 4.710 s

Internal trace

For the unequal-length dynamic batch:

encoder_out_lens:
[64, 73, 142]

alphas / pre_peak_index tensor width:
143

Before the fix, the timestamp helper received:

sample 0: 143 frames
sample 1: 143 frames
sample 2: 143 frames

For the runtime predictor:

CifPredictorV2
tail_mask=True
tail_threshold=0.45

the valid timestamp extent is the per-sample encoder length plus one CIF tail frame:

[64 + 1, 73 + 1, 142 + 1]
=
[65, 74, 143]

For the shortest sample:

143 - 65 = 78 extra frames

Plain Paraformer invokes the timestamp helper with upsample_rate=1, corresponding to approximately 60 ms per frame in this path:

78 * 60 ms = 4680 ms

The observed timestamp overshoot was approximately:

4710 ms

That was the first clue, but I also tested the mechanism directly.

---

3. Timestamp-only causal check

I kept the ordinary GPU dynamic batch unchanged.

The only behavioral modification was to trim the tensors immediately before timestamp conversion to the valid per-sample extent.

Recognition itself was not changed.

The timestamp-helper widths became:

before:
[143, 143, 143]

after:
[65, 74, 143]

The resulting timestamps changed from:

unmodified dynamic:
    max end = 24.710 s
    backward jumps = 1

to:

trimmed dynamic:
    max end = 20.030 s
    backward jumps = 0

At the same time:

ASR text:
    identical before / after

timestamp count:
    67 before
    67 after

This makes me suspect that the batch-maximum padded timestamp tensor extent is leaking into the timestamp conversion for shorter samples.

Current plain Paraformer passes:

ts_prediction_lfr6_standard(
    pre_peak_index[i],
    alphas[i],
    ...
)

without trimming those tensors using that sample's encoder_out_lens[i]:

funasr/models/paraformer/model.py

The timestamp helper then derives its frame extent from the received tensor:

funasr/utils/timestamp_tools.py

There is also precedent elsewhere in FunASR for trimming timestamp-predictor tensors using a per-sample encoder length before calling the same helper, for example in BiCIF Paraformer:

funasr/models/bicif_paraformer/model.py

---

4. Narrow patch that passed the real-model reproduction

The smallest patch I tested successfully was local to the Paraformer timestamp call site.

The guard is deliberately conservative:

• trim only if tail_mask is True;
• use encoder_out_lens[i];
• add one frame only if tail_threshold > 0;
• preserve tail_mask=False;
• preserve predictors without a tail_mask attribute.

--- a/funasr/models/paraformer/model.py
+++ b/funasr/models/paraformer/model.py
@@
                     if pred_timestamp:
+                        timestamp_pre_peak_index = pre_peak_index[i]
+                        timestamp_alphas = alphas[i]
+                        if getattr(self.predictor, "tail_mask", None) is True:
+                            timestamp_len = int(encoder_out_lens[i].item())
+                            if float(getattr(self.predictor, "tail_threshold", 0.0)) > 0.0:
+                                timestamp_len += 1
+                            timestamp_len = min(
+                                timestamp_len,
+                                timestamp_pre_peak_index.shape[-1],
+                                timestamp_alphas.shape[-1],
+                            )
+                            timestamp_pre_peak_index = timestamp_pre_peak_index[:timestamp_len]
+                            timestamp_alphas = timestamp_alphas[:timestamp_len]
                         timestamp_str, timestamp = ts_prediction_lfr6_standard(
-                            pre_peak_index[i],
-                            alphas[i],
+                            timestamp_pre_peak_index,
+                            timestamp_alphas,
                             copy.copy(token),
                             vad_offset=kwargs.get("begin_time", 0),
                             upsample_rate=1,
                         )

The patch was applied to the installed FunASR source, tested from fresh Python processes, and then the original source was restored.

Real-model result:

Environment
- FunASR: 1.4.3
- PyTorch: 2.11.0+cu128
- GPU: NVIDIA T4
- ASR: HF paraformer-zh
- predictor: CifPredictorV2(tail_mask=True, tail_threshold=0.45)

20.000 s target crop

Unmodified dynamic:
- max timestamp end: 24.710 s
- backward jumps: 1

Patched dynamic:
- max timestamp end: 20.030 s
- backward jumps: 0

ASR text:
- identical before / after

Timestamp count:
- 67 before / 67 after

Timestamp helper widths
encoder lengths: [64, 73, 142]

before:
[143, 143, 143]

after:
[65, 74, 143]

Patched forced-single:
- max timestamp end: 20.030 s

---

5. CPU-only regression is also possible

The real-model reproduction requires the model downloads and a GPU, so I do not think it needs to become a CI test.

A small deterministic regression can exercise the real Paraformer.inference() call site with dummy encoder/predictor/decoder outputs and mock only the timestamp helper.

For example, with:

encoder lengths:
[4, 8]

shared padded predictor width:
9

the current call site produces:

tail_mask=True:
[(9, 9), (9, 9)]

After the narrow patch:

tail_mask=True, tail_threshold=0.45:
[(5, 5), (9, 9)]

while the conservative cases remain unchanged:

tail_mask=False:
[(9, 9), (9, 9)]

tail_mask attribute missing:
[(9, 9), (9, 9)]

tail_mask=True, tail_threshold=0:
[(4, 4), (8, 8)]

So this can be covered by a fast CPU-only, network-free regression rather than a heavyweight integration test.

---

6. Public reproducer

I put the complete reproduction in two notebooks:

• Clean / rerunnable notebook
• Executed notebook with outputs

The notebook includes:

• exact dependency setup;
• environment capture;
• pinned public AISHELL-4 input;
• input SHA-256 verification;
• deterministic CPU call-site regression;
• unmodified dynamic-batch reproduction;
• helper-width tracing;
• source patch application;
• patched dynamic reproduction;
• forced-single comparison;
• transcript/timestamp-count invariance checks;
• exact patch output;
• source restoration.

---

7. Scope / things I am not claiming

I would keep this fix narrow.

I have not established that the same patch is safe for:

• tail_mask=False;
• legacy predictors without an explicit tail_mask contract;
• EParaformer;
• English/BPE timestamp paths.

I also do not interpret successful execution of the speaker pipeline as proof that speaker-diarization accuracy is fixed. I did not measure DER.

Similarly, although automatically normalizing:

output_timestamp
return_time_stamps

into:

pred_timestamp

looks tempting for the first issue, I would not make that a broad Paraformer-family change without separate compatibility testing.

---

Questions for maintainers

The two main questions I have are:

1. For plain Paraformer, is pred_timestamp intended to be automatically enabled when AutoModel requests timestamps for the speaker pipeline?

   If yes, the first finding may be a straightforward flag-propagation bug.

2. For CifPredictorV2(tail_mask=True), is trimming pre_peak_index[i] / alphas[i] to encoder_out_lens[i] + tail_frame before ts_prediction_lfr6_standard() consistent with the intended timestamp contract?

   The real-model and deterministic regressions both support this narrow behavior, but I would prefer confirmation before treating it as a general upstream fix.

If these are better tracked separately, I am happy to separate:

A. Paraformer timestamp-request propagation

B. Paraformer GPU dynamic-batch timestamp padding

into distinct bug reports / PR scopes.

[LauraGPT]

Thank you for the unusually complete report and for separating the two mechanisms. I independently checked this against FunASR main@8c60d278379de0044a9211a04660c9eeba08c66c.

Both findings are reproducible:

1. The speaker pipeline requests timestamps through output_timestamp / return_time_stamps, while plain Paraformer.inference() gates its timestamp branch on pred_timestamp. The no-timestamp speaker/VAD fallback also exposes sentence instead of the documented text key. These are contract mismatches rather than an error in your invocation.
2. On an H100 with the pinned public AISHELL-4 crop from your report, the unmodified dynamic batch reproduced the same encoder lengths [64, 73, 142], shared predictor width 143, maximum timestamp end 24.710 s for 20.000 s of audio, 4.710 s overshoot, and one backward timestamp jump. That confirms that the batch-padded extent reaches timestamp conversion for shorter samples.

Please split these into two independent PRs if you are willing; that will keep review and rollback boundaries clear:

• A — speaker timestamp/schema contract: use output_timestamp only as a model-local fallback when pred_timestamp was not supplied, so an explicit pred_timestamp still wins. In the no-timestamp speaker/VAD fallback, add the documented text key while retaining legacy sentence compatibility.
• B — dynamic-batch timestamp extent: only when predictor.tail_mask is True, trim each sample's pre_peak_index and alphas to encoder_out_lens[i] plus one CIF tail frame when tail_threshold > 0, clamped to both tensor widths. Preserve the current behavior for tail_mask=False, a missing tail_mask attribute, and zero threshold.

Fast CPU-only, network-free regressions covering those contracts are preferable for CI; the real-model H100 reproduction can remain integration evidence. Please keep EParaformer, English/BPE timestamp behavior, and any broader Paraformer-family flag normalization out of these two patches unless independently reproduced.

Link both PRs back here. This confirms the two bugs and their reviewable scope, but each patch will still need its own exact-head tests and review before merge.