SGLang Integration

[merrymercy]

Nice project!

I believe this project can greatly benefit from https://github.com/sgl-project/sglang. You can try to use SGLang as a backend for local models.

• The fast JSON decoding feature can help you force additional constraints and probably help the nested JSON schemas. You can find the example here.
• The RadixAttention feature can help you reuse the KV cache for the shared prefix. You can find one example on parallel decoding here.

[varunshenoy]

This is next on the list! Huge fan of the work you guys have done and think the constrained sampling + KV cache reuse will be a game changer with Super JSON Mode.

[dailydaniel]

@varunshenoy Are you planning to add the ability to use it with the SGLang OpenAI like server?