How does the cross attention work?

[SinanAkkoyun]

Hello,
I am trying to understand the autoregressiveness of the decoder.
Regarding the audio feature cross-attention: Does it for every iteration of the for loop (decoder.py, line 595 _main_loop) take ALL audio features (xa) simultaneously for each cross attention decoding or does it only take specific samples?

I want to understand how parallel/serial the whole approach is. I highly appreciate any answer!

[SinanAkkoyun]

@jongwook I hope the ping is ok

Forget what I asked above, this is the right question:
Given the for loop is at position 2s of a 5s wav file, it takes all 5s audio_features for prediction at any point, right? So, does predicting the token at timepoint 2s also take future audio_features into consideration?
What would happen if the decoder only had access to the current audio_features?

[SinanAkkoyun]

Awesome, after we hopefully gather some more intel we could work on that, I've been working on it and an interface for some time, planning on PRing when it's done

Yes, masked attention will be necessary, but also will require retraining... Regarding the relative positional embedding, why would that be needed? The sin pos encoding does scale from the beginning of a sentence, no?

[atyshka]

I guess the question is how would you handle sequences longer than 30 seconds? I’d envision chopping off the previous audio so you have a constant window of 30s audio. But if you do that, the sinusoidal position embeddings no longer start at 0… I’m not sure if that would be a problem or not

[SinanAkkoyun]

That's no problem. No single sentence is more than 30 seconds long. My plan is to do VAD to detection and reset the positional embeddings for every sentence. Contextual robustness of that level (inter sentences context is negligible for real time detection)

[SinanAkkoyun]

The bigger problem lies within the architecture and retraining... The whole decoder cross-attention needs to be replaced and the decoder retrained...

[atyshka]

The encoder also needs to be retrained, right? But yeah, the bigger problem is there's no code provided for training from scratch, and I'm not sure how many GPU-hours a model like this would take to train.

[SinanAkkoyun]

@jongwook Are there any resources on how to retrain the decoder? (instead of fine tuning it) Or do you have an idea on how to solve the above without major retraining?

[jongwook]

The decoder is autoregressive while the encoder is not. Encoder needs only one forward pass, so running the encoder on every slice using a sliding 30s window (with ~1s stride) is not a big performance hit, relatively. More disciplined way is use a Transducer model like this, but it'd be an entirely different architecture and probably requires training from scratch.

[SinanAkkoyun]

I see, thank you very much. You mean different architecture than whisper or than the nvidia model?

[jongwook]

The linked nvidia model is an example of transducer models, which works differently (while still being transformer-based) from Whisper, which is an encoder-decoder model. This blog article is a great introduction on how transducer models work.