Minimal glibc requirement for Meilisearch

[curquiza]

Last week, we had an issue with Meilisearch (v0.29.0) that didn't work on Debian 10 anymore, see meilisearch/meilisearch#2850.

The problem came from a simple reason: we updated our GitHub CIs from ubuntu-18.04 to ubuntu-20.04 because ubuntu-18.04 is currently deprecated and will be removed on 1st April 2022, see actions/runner-images#6002.

This made Meilisearch not work anymore with glibc 2.28, with a minimal requirement of glibc 2.29, not compliant with Debian 10.

So, we had to release Meilisearch v0.29.1, so Meilisearch is compatible again with Debian 10.

The problems

• We are currently not transparent about the minimal glibc and distributions Meilisearch needs to run, for example, it could be specified in the documentation.
• If we keep using native GitHub action, we won't be able to support Debian 10 on 1st April 2024. Unfortunately Debian 10 EOL is June 2024, see this link. However, it's possible to have our own GitHub runners even if it's not something we could have to run our tests (would need to have at least 8 runners on our side to ensure being still reactive merging PRs).

The discussion here is to decide together what we want to do regarding the minimal requirement of Meilisearch about the glibc.

I ping here: @alallema @brunoocasali (for integration team), @meilisearch/cloud-team and @Kerollmops and @irevoire (core team)
First, do you have any opinion regarding this?

[gmourier]

Thanks @curquiza

[brunoocasali]

Before further discussions, is it possible to embed the glibc into the final meilisearch build? I think this is one of the best options out there...

[curquiza]

Would be possible if I'm correct (@irevoire for confirmation), but the binary will be way bigger. In this case: should we keep releasing a non-embedded glibc binary? If yes, which binary do we promote first?

[brunoocasali]

I think for simplicity, we should only promote the one who has the embedded version because the user may don't know if they need or not to have the glibc included. So for me, it will be less error-prone to the users. 😄

[curquiza]

Do you have examples of applications/binaries doing this? Is it common? What are the cons doing this?

[brunoocasali]

I don't think it has a strong con besides increasing the final binary size. As far as I understand, the glibc has around 3MB installed I think this is a small price compared to the value it could bring.

I found this stack-overflow answer about the topic that could be related https://stackoverflow.com/a/32675381/2649707.

When you think of linkage with a static library, what happens is pretty clear: the linker copies the code from the library (e.g. libFoo.a) into your output binary. Your output file grows in size but doesn't need to resolve any external dependencies at runtime. Everything your program needs to run (with respect to the static library) is present after it is built.

With a dynamic library (.dylib, or system-supplied framework), the expectation is that the library you are linking against will be present somewhere in the system's dynamic-library loader path when you run your program. This way you don't have the overhead of copying all the third party external libraries into your binary, and all the different programs on a computer that also link to that library will be able to find it, which saves minimally disk space, but also potentially memory space, depending on how and where the system caches libraries.

Adding two more cents to his message: Since we cannot ensure anymore if we will have the glibc present or not, the dependency should be embedded in my POV. Otherwise, some particular configurations will not be able to run without dealing with those problems.

Do you have examples of applications/binaries doing this? Is it common? What are the cons doing this?

Unfortunately, I could not confirm if the binaries I remembered (eg. git, ) were using embedded libraries or not.

[irevoire]

Personally, I can't think of any good binary embedding the glibc in it; here is a random list of well-known binaries that don't do it:

• git depends on linux-vdso, libpcre2, libz, libc
• vim depends on linux-vdso, libm, libncursesw, libacl, libgpm, libperl, ld-linux, libcrypt, libc`
• curl depends on linux-vdso, libcurl, libnghttp2, libidn2, libssh2, libpsl, libssl, libcrypto, libgssapi_krb5, libzstd, libbrotlidec, libz, libunistring, libkrb5, libk5crypto, libcom_err, libkrb5support, libkeyutils, libresolv, libbrotlicommon, `libc
  And I'm going to stop here, but honestly for this question, no, it's not common at all.

What are the cons doing this?

One of the big cons seems to be that we would lose a lot of performances; we should measure it, though.

[curquiza]

After discussing with @irevoire and @brunoocasali, here some solutions we could suggest

Reminder: "not supporting glibc >= 2.29" meaning droping the current support of Debian 10 and before.

First solution: supporting glibc < 2.29 without musl binaries

We still compile "native" binaries (no-musl) with glibc 2.27

Pros

• we still support Debian 10 and Ubuntu-18.04 (and before), since the machine has glibc 2.27 on it.
• we don't have any loss of performance for the provided binaries

Cons:

• We cannot use the native GitHub runner anymore (stopping ubuntu-18.04 in April) and we have to configure our own runners, even for our tests; otherwise, the core-team doesn't make their implementation in a real production environment.
  • Either we configure a lot of self-hosted GitHub runners on our side to ensure we have enough velocity for our test suite -> can bring DevOps issues we don't anticipate yet
  • Or we run tests on ubuntu-20.04 when merging PRs and we run our tests every night on our self-hosted GitHub -> can create a gap and a lost of velocity during the sprint since we will notice bugs later and not when merging the PR
• We cannot ensure we support ALL the old Debian/Ubuntu versions because the current minimal glic version will still be glic 2.27

Second solution: supporting glibc < 2.29 with musl binaries

For people having machines using glibc >= 2.29, we provide a non-musl binary as we create them right now.
For people having machines using glibc < 2.29 (so Debian 10 and before), we provide a musl binary.

Pros

• Thanks to the musl binaries, Meilisearch can work on every Linux machine, not matter is the glibc on the machine.
• we don't have to change our processes and CIs to create binaries (cf cons of Solution 1 above)

Cons

• we need to update the download-latest.sh script provided in the docs accordingly -> not a big deal, but some additional work to do. Good point: thanks to this script, the "recommendation" will be transparent for the users: according to their machine, we will download the right binary.
• musl binaries can bring some performance issues -> we can also recommend compiling from the source for people who would like the best performance.

Third solution: not supporting glibc < 2.29

We officially drop the glibc < 2.29 support, so we drop the Debian 10 and Ubuntu-18.04 support.

Pros

• no additional process/CI or additional work to do

Cons

• we lose users of Debian 10/Ubuntu 18.04 and before, knowing both of these versions haven't reached their EOL yet.

---

What do you think @gmourier @brunoocasali @gmourier but mostly @Kerollmops and @dureuill?

[Kerollmops]

I would like @dureuill's comment on that 👀

[gmourier]

Do you know where we can find a recent usage rate as a server composed by version for Debian and Ubuntu?

I would have a hard time making relevant comments without this information.

[dureuill]

I'd like to propose a variant of solution 1: supporting glibc < 2.29 without musl binaries, but without custom runners.

GitHub allows running jobs in a container, so we could use the default Ubuntu 20.x runners but with a Ubuntu 18.04 container.

This is how rust-analyzer solved the issue for the time being, we can take inspiration from their release.yaml.

Pros

• we still support Debian 10 and Ubuntu-18.04 (and before), since the machine has glibc 2.27 on it.
• we don't have any loss of performance for the provided binaries

Cons

• needing to start a container might make the CI take slightly longer time

[curquiza]

I tested Louis' solution, and I can confirm I found a way to make it work and not change too much our CI.

I indeed cannot use the official Rust Docker image because they only provide debian 10 image, and debian 10 uses glic 2.28. And unfortunately only glibc 2.27 is available on ubuntu-18.04.

Luckily, to make it work, using ubuntu-18.-4 docker image, only installing curl and cc is needed in the container, and only adds 30second to the total CI 🎉

See the final solution I will implement in our official CIs (here it's just for test)
https://github.com/curquiza/meilisearch/blob/cab056bfddc40d9b30473c3038a5bf76dc63005e/.github/workflows/publish-binaries2.yml#L52-L77

🎉 🥳 So I confirm, Meilisearch will work on ubuntu-18.04 and later, and any compatible Linux distribution using glic 2.27 and later. 🥳 🎉

[dureuill]

Thank you so much curqui for this work on the CI 🚀, I think it is very important for us to support older distros when we can, as people don't always have a way to update their distribution.

[gmourier]

Hello everyone 👋

Thank you for all the exploration work that has been done here 🙇‍♂️

Since it has been solved, I'm locking that discussion.

Please open a new discussion if you have feedback to share or a new feature request in mind.

Thank you 🙇‍♂️