Is it possible to remove kv_cache entirely?

[catalwaysright]

I want to get rid of the kv_cache in whisper, including install_hooks for decoder. Is there some way I can do this? Any clue?

[jianfch]

You can disable it by commenting out all the lines except the return line in PyTorchInference.logits like this:

    def logits(self, tokens: Tensor, audio_features: Tensor) -> Tensor:
        # if not self.kv_cache:
        #     self.kv_cache, self.hooks = self.model.install_kv_cache_hooks()
        #
        # if tokens.shape[-1] > self.initial_token_length:
        #     # only need to use the last token except in the first forward pass
        #     tokens = tokens[:, -1:]

        return self.model.decoder(tokens, audio_features, kv_cache=self.kv_cache)

[catalwaysright]

Then the output would be wrong since we don't have install_hooks() to store the previous K and V.

[catalwaysright]

@jianfch According to my experiment, we need at least self.model.install_kv_cache_hooks() to make it run properly.

[jianfch]

Did a quick test and got the same results. The kv_cache stores the computed kv values of the previous predicted tokens. If you don't install the hooks, those previous values will just be recomputed (which is wasting compute because they are the same set of values).

[catalwaysright]

Oh you mean we take the whole tokens as input, then it makes sense. Thanks!