Ship Nitro as a Binary

[tikikun]

Nitro should be statically built and distribution as a binary

Tasks

• Build drogon with llama cpp
• Spin up Mac VM for testing
• Target mac os, x86, metal supported binary
• (clarify?) We have llm endpoint using ggml
• Server can be configured using a config file

Success criteria

• Nitro is multi platform binary
• Runs a Drogon C++ server
• Serves llama-cpp for Metal or CPU only modes
• Include encoding / decoding
• An architecture diagram showings whats up

[0xSage]

@tikikun can you elaborate a bit more on your issues please.

Template inspo: https://github.com/pytorch/pytorch/issues/new/choose

[tikikun]

Outcome:

• Have static build and single binary file (or multiple files) to run nitro

Problem:

• Currently nitro needs to be shipped with a container, which is huge in size (2.89gb)
• Nitro can be much lighter with just a single binary, but the way we code it now does not allow for static build.