Similarities should round the length up [LUCENE-8053]

[asfimport]

The encoding that we use for lengths currently rounds down in case the length cannot be stored accurately. We should round up instead so that frequencies can never be larger than the length.

---

Migrated from LUCENE-8053 by Adrien Grand (@jpountz)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

frequencies can always be larger than the length because of synonym stacking (discount_overlaps) so I'm not sure its worth doing this?

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

True, though it requires that the same token appears twice at the same position, which is usually not the case? Even though I agree similarities need to be able to deal with this case, I was more wondering wether some impls might degrade in quality if some terms have more occurrences than the document length. For the record, we used to round up before 7.0 since we actually rounded down 1/sqrt(len). Happy to close if we think this is a non-issue.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Doesn't need to be the same token: discount_overlaps (which is enabled by default) means that all tokens with posinc=0 are dropped from the length.

I don't think we should close the issue because it would be nice for sim to not have to deal with this case, i just think it wouldn't help unless we also removed discount_overlaps completely, so that length always "makes sense". We could try benching this across all of our sims again, maybe its really not needed. But I am not confident this is the case, last time i checked it was important because there are a lot of cases where "artificial" tokens are added (e.g. WDF/commongrams/etc) and this prevents skew. See #9048 for more details.