epic: Cortex is compatible with Ubuntu 20.04 (removal of GLIBCXX_3.4.30 and `CXXABI_1.3.13'

[AspenEyers]

Goal

• Cortex should be able to run on Ubuntu 20.04
• Ubuntu 20.04 has not reached EOL (April 2025), and there are still large pool of users on it

Tasklist

• Compile to diff target, without dependence on GLIBCXX_3.4.30 and `CXXABI_1.3.13'
• Otherwise, articulate why we should not do this (e.g. loss of functionality, etc) and update Cortex.cpp: Hardware & OS Compatibility Matrix [WIP] #1125 and Cortex.cpp: Model Compatibility Prediction #1130

Original Post

Subject: bug: version GLIBCXX_3.4.30' and CXXABI_1.3.13' not found

Describe the bug

Jan uses GLIBCXX_3.4.30 which is a symbol that doesn't exist for standard packages on ubuntu 20.04 libstdc++.so.6.

An extract from the error logs:

jan-data-folder/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by jan-data-folder/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro)

janhq/jan#1732 appears to suggest that the solution is to upgrade to a later version of libstdc++.so.6, which Is a solution I'm not particularly happy with, considering it's a reasonably core package that will make anything I build incompatible with most ubuntu 20.04 systems.

Is there any chance you can compile without these symbols?

Steps to reproduce
Steps to reproduce the behavior:

1. Go to https://jan.ai/
2. Click on download AppImage
   3 . I downloaded Mistral Instruct 7B Q4
3. write any instruction and check the error logs when it fails.

Similarly I tried the deb package:
https://github.com/janhq/jan/releases/download/v0.4.12/jan-linux-x86_64-0.4.12.AppImage

/home/aspen/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/aspen/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro)
/home/aspen/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /home/aspen/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cuda-12-0/nitro)

Environment details

• Operating System: Ubuntu 20.04
• Jan Version: 0.4.12

[the-vindicar]

This bug is critical, as it prevents the use of core functionality. Additionally, upgrading libc requires root privileges, and if you lack them, convincing sysop to perform the upgrade is not easy for reasons stated in the post.
Is there a version of Jan that can be run on Ubuntu 20.04 without modifications?

[imtuyethan]

[dan-homebrew]

janhq/jan#3489 is likely linked to this as well

[dan-homebrew]

Changing the Issue title to reflect error message.

• Jan is currently compiled for e are targeting later Ubuntu versions (>20.04)
• Need to decide if we should target earlier versions
• Either that: detect old GCC version, and show error message (e.g. Jan won't work, you need to upgrade X)

[dan-homebrew]

@0xSage This bug needs a decision on #1125.

• Should we support earlier versions of Ubuntu? (or just have a FAQ that asks users to update glibc?)
• Some enterprise users will not be able to update glibc (see comment from @the-vindicar above)

[dan-homebrew]

Transferring this to Cortex.cpp repo

[vansangpfiev]

We are using gcc-11 on ubuntu (20.04 for cortex.cpp, cortex.llamacpp; 22.04 for cortex.tensorrt-llm) to compile our sources.

• gcc-11 is mandatory for cortex.tensorrt-llm
• we can use gcc-9 for cortex.cpp and cortex.llamacpp

There are a few things I think need to be considered:

• Need to rebuild all the runners for cortex.cpp and cortex.llamacpp to use gcc-9 (cc @hiento09)
• Users can only use llamacpp engine on Ubuntu 20.04, to use tensorrt-llm engine they need to upgrade glibc
  cc: @dan-homebrew

[dan-homebrew]

We are using gcc-11 on ubuntu (20.04 for cortex.cpp, cortex.llamacpp; 22.04 for cortex.tensorrt-llm) to compile our sources.

• gcc-11 is mandatory for cortex.tensorrt-llm
• we can use gcc-9 for cortex.cpp and cortex.llamacpp

There are a few things I think need to be considered:

• Need to rebuild all the runners for cortex.cpp and cortex.llamacpp to use gcc-9 (cc @hiento09)
• Users can only use llamacpp engine on Ubuntu 20.04, to use tensorrt-llm engine they need to upgrade glibc
  cc: @dan-homebrew

@vansangpfiev @hiento09 Let's support gcc-9 as a default for now, and we can reconsider after its EOL in April 2025.

• I suspect there will still be a lot of Ubuntu 20s in the wild for +1 year after that
• Enterprises will likely be stuck on Ubuntu 20 for a while

I will create a separate ticket to figure out the UX for upgrading glibc for TensorRT-LLM, and schedule it with the TensorRT-LLM stories.

[gabrielle-ong]

I'm moving this to sprint 22 for QA

[gabrielle-ong]

Cortex can run on Ubuntu 20