[Grant request/project proposal] Training an open source MoE LLM for local deployment with weights offlading and disk streaming

[view321]

TL;DR
I want to train two opeb source MoE models targeted at disk streaming (first model 10000 USD, second model 80000 USD if the first succeeds, all in compute). I have a 200 dollar demo with promising results on small scale

IDEA
Two models targeted at disk streaming - 7B-A1B for ensuring the architecture and the training pipeline is sound. Then 24-28B total, ~2.5B active params model for fast (at least 20 tokens/second) inference on ~1000 dollar gaming laptops(16GB RAM, ~4GB VRAM), refered to as 'target device'

WHY NOT Bonsai-27B (similar size, good quant)
Bonsai 27B is a great model, but it would be slow on the target device, because every token touches the whole 4-6 GB per token (dense model). Ours would touch 1.2GB per token (much faster)

DEMO
Engine https://github.com/view321/nanocolibri-engine
Model https://huggingface.co/vovaRL/NanoColibri-Instruct
Training code
https://github.com/view321/NanoColibri

COST
10000 USD for the 7B model in compute tokens. It includes pretraining, SFT and RLVR/GRPO. So the full pipeline, to derisk the bigger model
About 80000 for the bigger model, 24 to 28B total params (final size and other details are decided after the first model's full results).
Also a stipend is probably needed (1500 USD per month)
6-8 month for the whole project, with every step open-source and documented

Everything above is open to criticism and discussion! Would be glad to get feedback