htsglang: heterogeneous, tier-aware sglang — uneven compute + VRAM pooling, session KV spill to system RAM, adaptive drafter routing, and improved GGUF support

[efschu]

https://github.com/efschu/htsglang/tree/docs/features-vs-upstream

Adaptive drafter routing switches the speculative draft model at runtime — by context length (policy mode) or measured acceptance (bandit mode). And more

[efschu]

now with gguf

[noonghunna]

this is amazing work man! thank you for sharing this.

[efschu]

Still some work in progress. ;)

[efschu]

https://github.com/efschu/htsglang/blob/docs/features-vs-upstream/FEATURES_VS_UPSTREAM.md
https://github.com/efschu/htsglang/tree/docs/features-vs-upstream

Still some quests to complete

[noonghunna]

you've found an pretty interesting direction i must say. 👍

[tomByrer]

Thanks! I had SGLang bookmarked since I have RTX3090 & 10GB RTX3080 on same machine.
Will your fork work with that?

note on GGUF + MTP throughput

I think MTP benefits really depends on hardware & use case.

[efschu]

Will your fork work with that?

Yes sure, it already does

I think MTP benefits really depends on hardware & use case.

Thats correct

[efschu]

#845