Skip to content

Repetition with canary-1b? #8776

Answered by titu1994
deklanw asked this question in Q&A
Apr 1, 2024 · 1 comments · 5 replies
Discussion options

You must be logged in to vote

Canary is an AED model, like Whisper. It is not trained with an alignment loss (CTC, RNNT, TDT) but next token prediction. So the decoder model, which has never seen text longer than 30-40 seconds, loses attention tracking once it goes to 1-2 minutes.

Replies: 1 comment 5 replies

Comment options

You must be logged in to vote
5 replies
@deklanw
Comment options

@titu1994
Comment options

Answer selected by deklanw
@deklanw
Comment options

@sukeyxu
Comment options

@kunibald413
Comment options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
4 participants