Currently the inverted index does not have segments, it has partitions. However, ignoring that for the moment, the inverted index also ignores the num_indices_to_merge argument. Every time we call optimize we just add a new segment (partition). As a result, if an FTS index is built incrementally, through many optimize calls, the resulting index will have many segments and the search performance will suffer. We should do something similar to the vector index, where we merge segments (don't add new ones) until a segment is large enough to stand on its own.
Currently the inverted index does not have segments, it has partitions. However, ignoring that for the moment, the inverted index also ignores the
num_indices_to_mergeargument. Every time we call optimize we just add a new segment (partition). As a result, if an FTS index is built incrementally, through many optimize calls, the resulting index will have many segments and the search performance will suffer. We should do something similar to the vector index, where we merge segments (don't add new ones) until a segment is large enough to stand on its own.