[Feature]: Benchmark truncated vector size

[acidjunk]

Contact Details

No response

What should we build?

For some models we truncate the vector size on the assumption that the first part contains the most important context. A benchmark, with some known cases might be handy to remove the assumption.

Relevant pseudo code

[timfdev]

This is the same as the E2E test ticket right? If we have a benchmark dataset with stable ranking, we need a record function to upate indexed data anyway.

For testing truncated embeddings we can use the same mechanism.

[acidjunk]

No not completely the same.

This is a benchmark te determine what the effect is of truncating the vector size.
But I think you're right: the mechanisme is the same.

Probably we run, te suite, once with truncated vectors -> when results are ok -> done.

The other tests should be executed after changes to find regressions in search functionality.