Kimi with Colibri, environment variables, from newbie to newbie

[Limalski]

This is not instruction, it is more like "this-works-for-me".

I got personally quite frustrated what parameters to use at compile time, what parameters to run. At first I thought I could use colibri + CUDA + Kimi, but that did not seem to work and some friendly guys here instructed me to use Vulkan instead.
I can still see, there is mentions in CUDA in Kimi_k3.c.

So with these parameters I get about 20-55% GPU usage (VRAM usage still about 4500 MiB), but at least it works. I baked all in, and I still can't understand why the "serve" doesn't work. I got it working already at somepoint, but now it doesn't work again.

Basically I just bake all in, since I don't have any clue what is the compiler directive ifdef and what is environment variable for running binary. Help me out, if you know better.

"This works for me"

1. Cloning repo:
   git clone https://github.com/JustVugg/colibri

2. Switching to development branch
   cd colibri
   git switch dev
   git branch --show-current #to verify

3. patching with thread-tuning (Thanks to Streaml1ne J. Moose)
   colibri-thread-tuning.patch
   patch -p1 <colibri-thread-tuning.patch

4. Compiling Colibri
   cd c
   K3_VK=1 VK=1 CUDA=1 K3_CUDA=1 COLI_CUDA=1 ./setup.sh

5. Compiling Kimi-k3
   make -C . CUDA=1 K3_CUDA=1 COLI_CUDA=1 kimi_k3

6. Running Kimi with Colibri
   VK=1 K3_VK=1 CUDA=1 K3_CUDA=1 COLI_CUDA=1 COLI_TEMP=0.65 NUCLEUS=0.9 K3_THINK=0 OMP_NUM_THREADS=5 K3_MAXT=16384 K3_VK_GB=11 K3_VK_UP=10 K3_EXPERT_GB=10 c/coli chat --model /kimi --ram 80 --ngen 10000 #try 237

Own notes/disclaimer:

1. I have no idea if Mr. Mooses patch speed up or not. There are so many unsure and system specific things with this project at least for me.
2. I know I try to use CUDA and Vulkan at same time. I don't know why, but it works for me. If you know better, let me know
3. Despite I specify --ram 80, it does not work like that. RES of my top listing shows "46,2g"
4. I don't know why vram is not fully allocated, I made another issue about that. It could have something to do with ReBAR, aka. Resizable BAR. That would require me to update my bios, since the option is not visible for me.
5. I know some of my options are for compiling and some are environment variables for running only. What can I say, life is too short to figure out which is which.

This is not instruction, it is more like "this-works-for-me".

[Limalski]

note to myself.

building from main tree,

Cloning repo:
git clone https://github.com/JustVugg/colibri
cd c
COLI_VULKAN=1 COLI_CUDA=1 ./setup.sh
make -C . COLI_VULKAN=1 kimi_k3

running:
VK=1 K3_VK=1 CUDA=1 K3_CUDA=1 COLI_VULKAN=1 COLI_CUDA=1 COLI_TEMP=0.65 NUCLEUS=0.9 K3_THINK=0 OMP_NUM_THREADS=5 K3_MAXT=16384 K3_VK_GB=11 K3_VK_UP=10 K3_EXPERT_GB=10 c/coli chat --model /kimi --ram 80 --ngen 10000 #try 238, probably still too many useless environment variables...

[Limalski]

note to myself. Better to use dev branch:
"This works for me"

Cloning repo:
git clone https://github.com/JustVugg/colibri
cd colibri ; git log
git switch dev; git log
cd c
make -C . kimi_k3
cd ../web
npm install ; npm audit fix ; npm run build
cd ../c
./setup.sh
cd ..
COLI_RAM_OVERCOMMIT=1 K3_MAXT=16384 c/coli web --ram 60 --model /kimi --ngen 10000

This way I can get web page open and Kimi works. I just bought new computer, need to optimize further.
Add Vulkan and Cuda for compiling maybe, but no cuda for running.