epic: Structure Manual QA for cortex.cpp

[dan-menlo]

Goal

• cortex.cpp

Tasklist

• Build up Manual QA checklist for cortex.cpp
  • Hardware compatibility matrix
  • Should cover key stories
  • Should include column that tracks "automated testing" (i.e. is there automated tests to cover)
• Create releases folder in cortex.cpp, where the QA checklist should be committed together with key learnings, lessons to preserve for younger engineers

[dan-menlo]

@gabrielle-ong For v1.0's Manual QA, I would like us to focus on answering the following questions:

1. How do I provision all Hardware/OS types to do the testing
2. Does the Installer install the correct files to their respective locations?
3. Does the Installer install the correct version of llama.cpp?
4. Is the user able to pull, run and chat with llama3.1 successfully?
5. Does the Uninstaller uninstall all files, and not leave any dangling references?

I provide more context and links to key issues below:

Provisioning of Hardware/OS

Most of our bugs come from when cortex.cpp breaks on some hardware/OS. Catching these will reduce the number of bugs in Jan.

• We have invested in internal hardware that covers most critical/common hardware and OS types
• We should be able to provision clean VMs to test Jan for QA (speak to @hiento09)
• For each release, we should QA this cortex.cpp on all hardware and OS that we are compatible with
• Cortex.cpp: Hardware & OS Compatibility Matrix [WIP] #1125

Additionally, I would like to document our process provisioning a VM. This may grow to become its own product in the future (e.g. Menlo Cloud), as other teams may also need cross-platform testing.

Testing Installer

Installers are OS-specific, and install a clear set of files for each operating system. Installers will also need to detect hardware and pull the correct version of llama.cpp (I will cover that in the next section).

First, we will need to verify that the Installer for a particular operating system writes the correct files to the correct locations. The following issues/discussions have relevant context:

• Cortex.cpp: Installation & Uninstallation #1114
• epic: Cortex.cpp Installer MVP (Local & Network Installer) #1030

Testing Hardware Detection and llama.cpp binary

Installers will also detect the user's hardware, and then pull the correct version of llama.cpp. This is based on:

• Operating System (e.g. Windows, Mac, Linux)
• CPU Extension (e.g. AVX-2, noAVX, AVX-512)
• GPU Acceleration (e.g. CUDA11, CUDA12, Vulkan, sycl, etc)

We should be aligned with the versions published by llama.cpp: https://github.com/ggerganov/llama.cpp/releases

We need to verify that an AVX-2 system is correctly identified as an AVX-2 system, and the correct version is pulled.

We have an open issue, discussing how llama.cpp is installed, which may change how we QA this:
#1217

Pulling and Running Models on llama.cpp

We should verify that Pulling and Running Models writes to the correct Model Folder, and with model.yaml. The following issues/discussions have relevant context:

• epic: Implement new Model Folder and model.yaml #1154

Testing the OpenAI-compatible API, and CLI commands

We should verify that key API endpoints work:

• Postman collection testing (e.g. manual testing)
• Later on, the eng will likely implement some automated tests using the OpenAI library (e.g. for /chat/completions)
• The key things to verify are the non-OpenAI endpoints, e.g. model loading, model starting, etc

We should also verify that key CLI commands work:

• e.g. cortex run, etc

Uninstallation

We should verify that Cortex's uninstaller removes all relevant files, and does not leave dangling references.

Updater

We should verify that Cortex's updater works, but this may be challenging for now.

[freelerobot]

Manual checklist template:

• https://www.notion.so/jan-ai/QA-Cortex-Sprint-20-1065b91df6b1809bbb3cdb153d5ed14e?pvs=4

Bugs

• bug: cortex update stdouts "Update cortex sucessfully" even when update failed #1257
• bug: cortex-nightly -v shows v68 after upgrading from v66 -> v67 #1259
• bug: cortex engines get is broken #1266
• bug: updater is broken for v67, 68 #1264
• bug: cortex pull author/model_id && cortex pull model_id returns "Model not found" #1263
• bug: cortex pull https://huggingface.co/cortexso/mistral doesnt work & has buggy logs #1269
• bug: cortex pull invalid_url creates a model folder #1270
• bug(mac): cortexcpp folder installs with wrong permissions #1274
• bug: cortex run llama3.1 is broken #1273
• bug: cortex run flakiness #1272

Not QA'ed

Issues that are still in progress:

• model.yaml structure
  • Atm, downloaded models duplicate yamls in /model_id folder and in /models, whcih will likely be fixed

Questions

• idea: A subcommand is required should just stdout -h #1262
• discussion: --verbose flag #1261
• question: when do we write into /logs? #1265
• question(flaky): cortex -v hangs when checking if there is a newer release? #1268

[freelerobot]

Nonurgent Question @vansangpfiev @namchuai :

1. Where do I view the new API?
2. Do we autogenerate the API reference documentation?
3. I'd like to QA for OpenAI compatibility at some point. Should we QA API in this milestone?

[freelerobot]

I itemized test cases here: #1147