Handling varying milli versions/dumps

[GregoryConrad]

Related: meilisearch/meilisearch#2570

[GregoryConrad]

Initial Thoughts

Here are some initial thoughts based on preliminary research.

Dumps

Unfortunately, it appears dumps are not a good option.

The way meilisearch itself handles dumps is subpar (to say the least). However, this is not their fault; the problem is rooted in the fact that different versions of dumps require different features/functionality. Thus, many dump conversion utilities are needed to read older dump versions (i.e. v1_to_v2, v2_to_v3, etc.).

Further, we do not know when a user of this package will update this package. Thus, it is impossible to know on a user's device when we should make a dump, which means you would have to make a dump after every write transaction. At that point, you might as well just have a mirrored data store of all documents using a different database.

Multiple Milli Versions

Packaging multiple versions of milli in this package seems like the current best option.

The idea: only include newer versions of milli (that break index format) into the project as required to reduce needed dev work and plugin size. If index format between milli versions is the same (perhaps for small releases or bug fixes), then we can simply do an update of the milli package version to include the new version (e.g. replace "=0.35.0" with "=0.35.1" below)!

Then, this package itself will have a reference to every milli version supported and its corresponding wrapper, so we can easily migrate between versions at-will. This should be possible with something like the following in Cargo.toml:

[dependencies]
milli_v1 = { package = "milli", version = "=0.35.0" }
milli_v2 = { package = "milli", version = "=0.36.0" }
milli_v3 = { package = "milli", version = "=1.0.0" }
# ...

We will also need to implement a wrapper for each version of milli; however, this should not be too much work as the milli APIs probably won't change that much. Mostly just a copy-paste.

With this approach, we will also need a milli_instance_settings.json file in each instance directory to store the current version of milli of each index so migrations can be done.