cdc's old value cache sometimes exceeds the quota

[YuJuncen]

Bug Report

What version of TiKV are you using?

v6.5.1

What operating system and CPU are you using?

Not sure, but probably irrelative.

Steps to reproduce

(Not checked, but this did be observed in a production cluster.)

1. Create a table with small raws(perhaps with a sole column of primary key), let it be t1.
2. Create a table with larger raws(perhaps with a huge blob or json column), let it be t2.
3. Open a changefeed which observes both t1 and t2.
4. Update as many different rows as possible in t1.
5. Update as many different rows as possible in t2, until the memory usage of old value doesn't increase any more.

What did you expect?

After step 5, the memory usage of old value cache should be kept under the memory quota.

What did happened?

The old-value cache usage exceeds the quota.

[YuJuncen]

I have reproduced this by a unit test case:

#[test]
    fn test_old_value_capacity_not_exceed_quota() {
        let mut cache = OldValueCache::new(ReadableSize(1000));
        fn short_val() -> OldValue {
            OldValue::Value { value: b"s".to_vec() }
        }
        fn long_val() -> OldValue {
            OldValue::Value { value: vec![b'l'; 1024] }
        }
        fn enc(i :i32) -> Key {
        Key::from_encoded(i32::to_ne_bytes(i).to_vec())
        }

        for i in 0..100 {
            cache.insert(enc(i), (short_val(), None));
        }
        for i in 100..200 {
            // access the previous key for making it not be evicted
            cache.cache.get(&enc(i-1));
            cache.insert(enc(i), (long_val(), None));
        }
        assert!(cache.cache.size() <= 1000, "but it is {}", cache.cache.size());
    }

It panicked at:

thread 'old_value::tests::test_old_value_capacity_not_exceed_quota' panicked at 'but it is 28647'

Which means it exceeded about 28x of the quota.