keep using LittleEndian if hnsw+pq for backwards compatibility

[abdelr]

What's being changed:

Reverting the use of BigEndian in case of HNSW+PQ for backwards compatibility.

Originally, we used LittleEndian everywhere. Then in 1.23 we introduced flat and flat+bq and we changed how the Ids are stored and used BigEndian instead, so we could seek in the files using the Ids correctly. This was necessary for the filter logic. Later, when we unified, we introduced the bug since hnsw+pq was originally using LittleEndian for the Ids. Now, for backward compatibility, we cannot use BigEndian for the Ids in the hnsw+pq case, but the rest should be fine.

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline:
• All new code is covered by tests where it is reasonable.
• Performance tests have been run or not necessary.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[jeroiraz]

given both compressors (PQ and BQ) may be using the same bucket name, is it already in place a validation to detect which compressor is being used? so to handle a change in settings

[trengrj]

Do we also need to keep LittleEndian for flat+bq compatibility?

weaviate/adapters/repos/db/vector/flat/index.go

Line 202 in 5a02ebb

binary.LittleEndian.PutUint64(slice[i*8:], vector[i])

[abdelr]

given both compressors (PQ and BQ) may be using the same bucket name, is it already in place a validation to detect which compressor is being used? so to handle a change in settings

Currently, we do not allow switching from one compression method to the other. No need for handling config updates in such sense. When resuming the server though, we do need, but we have indeed a mechanism to detect the proper compression method.

[abdelr]

Do we also need to keep LittleEndian for flat+bq compatibility?

In the case you pointed out, it is the encoding of the vector values, not the ids. I think, we have not changed anything in this part. Check here the compressor code is using also LittleEndian

[abdelr]

I have updated the description for better understanding.