h2o is not working when the input is short.

[mutonix]

When I run bash scripts/streaming/eval.sh h2o, I find h2o is not working and acting exactly the same as the full. When I check the code

H2O/h2o_hf/utils_real_drop/modify_llama.py

Line 68 in 8bb3fe1

def __call__(self, past_key_values, attn_score_cache):

    def __call__(self, past_key_values, attn_score_cache):

        self._update_hh_score(attn_score_cache)

        if past_key_values is None:
            return None
        seq_len = past_key_values[0].size(self.k_seq_dim)
        if seq_len <= self.cache_size:
            return past_key_values

, I find the program always falls in the line seq_len < self.cache_size and directly return without running the following h2o code. However, the cache_size is set to 2048 as default. Isn't that a little long？

Or we can consider that h2o will not work in the short context?