Whisper misses the 2nd speaker

[Majdoddin]

On short audios (<30s) with two speakers, if the first speaker speaks less than 15s, whisper misses the second speaker, transcribing just the first one. I have observed it just with 'without_timestamps=False'.
This happens about 1/3 of the time.
This can be avoided sometimes, by inserting a silent spacer >= 1s between the two speakers parts.
See THIS notebook for a demonstration.
@jongwook Any ideas?