Welcome to colibrì Discussions — ideas, benchmarks, and Q&A live here now

[JustVugg]

Welcome! 🐦

Colibrì went from a personal experiment — can a 744B MoE run on a 25 GB consumer box? — to a community effort in about a week, and the issue tracker has been doing three jobs at once: bugs, benchmarks, and research ideas. Discussions is now open so each conversation gets the right home:

• 💡 Ideas — research directions, "what if" proposals, papers worth stealing from. This community has already produced router-lookahead corrections (Two-step shared-expert router prediction: +3.1% recall over PILOT (experiment, with prior art) #200), co-activation disk layouts (Profile-guided expert layout + co-activation profiles — data from a GGUF-side experiment (mbolt) #119), grammar-forced drafting (Idea: grammar-aware draft source — forced tokens in constrained JSON output are free, guaranteed-accepted drafts (composes with DRAFT/MTP; engages even with the int4 head from #8) #48), and honest negative results (Cache-aware drafting: a measured null, plus draft-vs-verify disk-load instrumentation #182 — we treasure those). If you have a hunch, post it here before writing code; someone probably has a measurement that helps.
• 🙌 Show and tell — benchmarks and datapoints go here now (instead of Issues). Hardware, disk, config, numbers — same rigor as the README table: commit, exact commands, warm-up policy. The best ones still get a row in the table.
• 🙏 Q&A — setup help, "why is my tok/s low", platform quirks (Windows/macOS/ARM all work now, and all have their gotchas).
• 📣 Announcements — releases and big merges land here.

Issues stay for bugs and concrete work items — things with a reproduction or a diff.

House rules, unchanged since day one: measure before claiming, post the commands with the numbers, and retractions are a feature not a shame (ask anyone — half our best findings started as corrections). A 744B model on consumer hardware only stays honest if we do.

Introduce yourself below if you like: what hardware are you running colibrì on, and what would you build if the tokens were 10× faster? That second question is not rhetorical — it drives the roadmap.

[RonitBStudent]

Hey! I’m Ronit, a high school student interested in machine learning, computational biology, and running powerful models on accessible hardware. I’m currently experimenting with colibrì on my personal computer and am especially interested in benchmarking, optimization, and possible research applications.

If token generation were 10× faster, I’d explore building a local research assistant that could analyze scientific papers, compare methods across studies, and help generate and test computational research ideas without relying on cloud APIs. Excited to learn from the community and contribute any useful benchmarks or observations I find!

[JustVugg]

Welcome Ronit — though you hardly need the welcome: your #145 is already merged into main (the native-Windows platform-detection test), so you're not an aspiring contributor, you're a contributor. 🐦

Your 10×-faster answer is exactly the kind that shapes the roadmap — and here's the part you might not know yet: half of your research-assistant idea doesn't need the 10× at all. Paper analysis is a constrained-output workload, and that's where colibrì is already strongest: GRAMMAR= (grammar-forced drafts, docs/grammar-draft.md) and the new SCHEMA=file.json (JSON-Schema → grammar, #148) make structured extraction — methods tables, study comparisons, claim/evidence pairs — dramatically cheaper per token, because the structural tokens cost no expert reads. A local pipeline that reads a paper and emits a structured comparison JSON is buildable today at current speeds. If you build even a small version, post it in Show and tell — it would be the first real application datapoint, which is worth more to this project than another tok/s number.

For the benchmarking interest: #208 has the template, and A/Bs (one knob changed) are the most valuable kind. For optimization: #209 lists the open frontier — problem 2 (best-of container) has an engine-free measurement path that needs no special hardware, just rigor, which you've already shown you have. Glad to have you here.

[RonitBStudent]

Thank you, Vincenzo! I really appreciate the welcome and the merge of #145. I hadn’t connected grammar-forced drafting with structured research extraction, so that’s a really interesting direction.
I’m going to start with a small, reproducible prototype that takes a paper and extracts claims, evidence, methods, and results into a JSON schema. I’ll document the hardware, commands, and measurements, then share it in Show and tell once it’s useful. I’ll also read through #208 and #209, especially the engine-free path for problem 2.
Glad to be contributing—thanks for pointing me toward work that would be valuable to the project! 🐦

[RonitBStudent]

Quick update: I built the first version of the research-paper extraction prototype here:
https://github.com/RonitBStudent/colibri/tree/agent/research-paper-extractor/examples/research_assistant
It includes a structured paper schema, grounded-evidence checks, a synthetic sample, and a repeatable baseline-vs-SCHEMA benchmark harness. It uses only the Python standard library. The local tests pass, the schema compiles through Colibrì’s actual JSON-Schema→GBNF compiler, and make check passes.
My current machine does not have enough storage for the full checkpoint, so the remaining step is running the documented two-rung A/B on a loaded Colibrì rig. If someone can run the harness, I can analyze the reports and turn the results into the Show and tell application datapoint.

[WalkingDead1407]

Hey everyone! I'm Tooshar ,aka walkingdead, a 1st year college student with interests in machine learning, large language models and data science. I recently came across colibrì and was immediately intrigued by the idea of running models of this scale efficiently on consumer hardware. I'm looking forward to experimenting colibrì on my device and test how everything works.

If token generation were 10× faster, I'd love to build a local coding assistant capable of reasoning over large codebases, debugging complex projects, and handling long-running reasoning tasks that are currently too slow to iterate on locally. Faster inference would make it much easier to prototype new ideas, benchmark different approaches, and experiment with more capable agentic workflows.

Excited to learn from the community and hopefully contribute to it!

[andy12b]

Hi there,I'm Andrei (M.Sc. Engineer) I love the project and I have an idea. I know you guys are all about the SSD and no GPU involved, but it would be nice if you could add this as a plugin or option:

Speculative Decoding (Draft & Verify) using the GPU.

Most gaming laptops have a dedicated GPU (like my RTX 5060 with 8GB VRAM) that currently sits completely idle. If we could load a smaller draft model into the VRAM to instantly generate draft tokens, and use the giant MoE on the CPU/SSD purely for single-pass verification, it could easily maximize performance and give a massive 3x speedup. It would be the perfect way to fully utilize the hybrid hardware setup that most consumer boxes have!

PS: If you already worked on something like this, I would love to try your alpha or beta version if available :))

[JustVugg]

Welcome Andrei! Good instinct — and you're closer to the truth than you might think, with one economic twist that changes where the GPU helps.

Colibrì already does draft-&-verify natively: GLM-5.2 ships its own MTP head (multi-token prediction), and the engine drafts with it + verifies in one batched forward (2.2–2.8 tok/forward when the cache is warm, #8). The draft source is pluggable — we have three today (MTP head, n-gram, grammar-forced #48) — so a GPU-resident small draft model is a legitimate fourth source, and your idle 8 GB RTX 5060 could raise draft quality well above n-gram.

The twist: on this engine drafting is not the bottleneck — verification is. Every drafted token still needs the full 744B verify pass, and each rejected draft costs real expert reads from disk (measured: speculation on a cold cache is a net loss — ~660→1100 expert-loads/token, #8/#120). So a better draft source pays exactly in proportion to its acceptance rate on a warm cache, where the verify reads are cheap. A 1-2B draft model in VRAM plausibly beats the MTP head's acceptance — that's the number that would make the case.

If you want to prototype it: the seam is the draft-source interface in spec_decode (grammar and MTP both feed it), and the OpenAI server already speaks to external processes. Measure acceptance-vs-MTP on a warm cache and post it — if it wins, it's mergeable as an opt-in source. This is exactly the kind of idea Discussions is for.

[andy12b]

That makes total sense, thanks for explaining the verification bottleneck.

What if you can try a different angle to bypass the SSD limits? I was thinking about a cascade/pipeline setup.

Since GLM is natively Chinese, force it to do all the heavy reasoning strictly in Chinese, using super concise prompts with zero filler words. Chinese is incredibly token-dense, so it would drastically cut down the number of tokens the SSD actually has to generate.

Then, just pipe that short output to a small, fast model sitting on the idle GPU. It would instantly translate and expand the conversational text back into fluent English, while leaving any generated agentic code completely untouched.

Basically: the SSD handles the ultra-compressed logic, and the GPU does the heavy lifting for text generation. Do you think this could be a viable workaround ?

[andy12b]

Hey again,

After looking into it, I realized two main bottlenecks for the external GPU draft: HTTP overhead would completely kill the speculative decoding latency, and I absolutely need a 1:1 token ID match for the verification to work.

Here is the hybrid plan I'm going to prototype:

Vocabulary Match: I'll run a small GLM-family model (like GLM-4 1.5B or a heavily quantized 9B) entirely on the RTX 5060 VRAM. This ensures the draft tokens perfectly match the GLM-5.2 vocabulary.

Zero-HTTP Bridge (IPC): I'm skipping the OpenAI server route. Instead, I want to build a Shared Memory (IPC) bridge directly into your spec_decode interface in c/glm.c. The GPU will write the draft tokens to memory, and the C engine will read them instantly.

The "Chinese Compression" Pipeline: To help the SSD even more, I plan to prompt the main GLM-5.2 to do its reasoning strictly in Chinese (maximizing token density). The GPU process will then intercept it and expand/translate it to English, leaving any agentic code blocks completely untouched.

I'm spinning up an AI coding agent to help me build this C/Python shared memory bridge. I'll report back with benchmarks once I test the acceptance rate on a warm cache.

[QQSHI13]

Hi there, I was exploring the possibility of running other powerful MoE LLMs that have smaller active params per token so that they could run on my device with only CPU and 16GB RAM. I am not very in the specs, but it looks like colibrì can do it.

[dannythevibe]

It can actually, and I just got this working

I just pushed a PR that implements an I/O-Aware RAM Fallback Router directly into the engine's core C inference loop.
It completely bypasses the SATA/NVMe bandwidth bottleneck, so you can smoothly run the massive 744B model on just 16GB of RAM without locking up your OS. I'm just waiting on the repo maintainers to merge the Pull Request

[wzxzhuxi]

Hi, I'm Zhuxi, an embedded AI developer. This is truly a crazy idea—you've solved the operational issues, and now you want to tackle the problem of running speed. I'm developing on the RK3588 chip (with 8GB of RAM). Previously, I deployed a 7B DeepSeek model, and I think I can learn from colibri to try deploying even larger models. If we can solve the problem of slow inference speed, we could get rid of server dependence—that's incredibly exciting.

[dannythevibe]

Hi, I'm Daniel. This is absolutely a crazy idea and i came across it on instagram.

I had two ideas to make massive models run perfectly on constrained CPU-only laptops:

1. I/O-Aware RAM Router: A custom C-engine router that dynamically drops disk reads and falls back to RAM-resident experts the second a slow HDD starts thrashing.
2. Native Semantic Memory: A lightweight local text journal that replaces heavy RAG databases by doing instant, string-matched memory injections directly into the system prompt.
   I actually went ahead and wrote the code for both of these and I’ve just submitted a Pull Request with the implementation

[jeswr]

Hi all — we run a set of experiments on grounding and checking LLMs with a training-free "concept kernel" (concepts built from a small set of semantic primitives) plus a deterministic rules engine (github.com/jeswr/kernel-of-truth). Two questions drive the work: does grounding + deterministic checking add genuinely checkable value beyond a model's generic output, and can it make models cheaper to run? colibri is how we access GLM-5.2 (int4, CPU expert-streaming) for the efficiency side — thanks for building it.

One strand there, in the spirit of "what would you build if tokens were 10× faster": profile what each of GLM-5.2's experts actually specialises in, find the ones computing regular/deterministic functions, and swap those for small deterministic modules — some grounded in our concept kernel, some just plain code, whatever holds quality. Very exploratory right now.

A few early numbers (measured, not claims): we've reconciled the routed-expert count to 19,456 from the estate metadata (matching the Expert Atlas thread), and stood up a per-token router trace with per-item reset. Concept-shaped routing structure is clearly real (permutation p = 0.0001), but a kernel-guided routing gain looks small so far — a cluster-aware interval sits near zero, so "real structure, modest exploitable signal, more to measure." We'll come back with commands + numbers as a Show-and-Tell, measure-before-claiming style.

[JustVugg]

@jeswr this is exactly the kind of work I hoped colibri would enable — and the measure-before-claiming discipline is very welcome.

There is a concrete place where your expert-specialization signal could land immediately. We just merged EXPERT_BUDGET=N (#254): it caps distinct experts per layer to cut disk I/O, keeping the highest-aggregate-gate-weight ones and dropping the rest. On a cold/low-RAM host it is a big speedup (measured ~2.75x decode), but it drops experts blindly by gate weight, and at an aggressive cap it visibly degrades coherence — fluent output that drifts off-topic. That is the core tension: fewer experts = faster, but which ones are safe to drop?

Your strand — find experts computing regular/deterministic functions and swap them for small modules — is the quality-aware version of exactly that decision. If your router trace + specialization profile can label experts as "deterministic / low-information / safe to skip" vs "load-bearing," that turns EXPERT_BUDGET from a blunt gate-weight cut into a principled drop set — speed without the coherence collapse. Even a static per-layer skip-list would be directly consumable by the engine.

Two shared hooks: the Expert Atlas thread (#175) and c/tools/expert_atlas (probe harness, plus the 19,456 count you reconciled — matches ours). Both are aimed at exactly the "what does each expert do" question.

Please do bring the commands + numbers as a Show-and-Tell (#208). Even a null result on the routing-gain side is useful signal. If you want a per-token expert-selection trace from the engine to feed your analysis, STATS=<file> dumps the usage histogram and there is a per-position trace path — happy to point you at the exact env.

[awdemos]

Congrats on opening Discussions! I love the "measure before claiming" rule.

One thing I'd be curious to see in the benchmarks: total cost-of-ownership for a long Colibrì session versus a flat-rate hosted GPU box. Consumer inference saves on API tokens, but once you factor in power, cooling, amortized hardware, and the time spent tuning, the breakeven can be surprising—especially for teams that run agents for hours.

We run UltraWork (https://vibecodingagency.com/gpu-cloud/) on flat monthly GPU environments partly because of that math. A direct measurement from the Colibrì community would be more credible than anything I could post.

Disclosure: I help run Vibe Coding Agency. Not pitching against local inference—just adding a datapoint the benchmark thread could use.

[israelmarques1024-dotcom]

ola rapaziada, queria falar com devs mobile, to criando um projeto chamado karnel termux, ele adiciona tudo q vcs precisan para programar no cell, para mais detalhes: https://kerneltermux.vercel.app/