Support Qwen3 and Qwen3MoE

[bozheng-hit]

Adding Qwen3

This PR adds the support of codes for the coming Qwen3 and Qwen3MoE models. For information about Qwen, please visit https://github.com/QwenLM/Qwen2.5. @ggerganov

[ggerganov]

For information about Qwen, please visit https://github.com/QwenLM/Qwen2.5.

Don't see information about Qwen3 - maybe not published yet?

[bozheng-hit]

For information about Qwen, please visit https://github.com/QwenLM/Qwen2.5.

Don't see information about Qwen3 - maybe not published yet?

We’ll update the blog once the model is officially released—hopefully very soon!

[ShuhaibNC]

Qwen3 is the nextgen AI

[Dampfinchen]

Excited for it. Hope it has native multimodal support and a huge boost in creative writing (lacks in that department imo)

Anyways, kudos on implementing support so early! Others should take note.

[unclemusclez]

lol awesome

[ptSpec]

Good guy Devs, added day 1 support, Thank you!

[x0wllaar]

Excited for it. Hope it has native multimodal support and a huge boost in creative writing (lacks in that department imo)

Anyways, kudos on implementing support so early! Others should take note.

It most likely won't. In the transformers commit they had no processors, and if I understand correctly, no vision here. We'll have to wait some for vision modules to be integrated

[red-co]

Hopefully it will be a native lean causal inference model rather than a bloated multimodal model,

[ngxson]

🔥 day-0 support for Qwen3 + Qwen3MoE, looking forward to the release of the weight!!

[Dampfinchen]

Hopefully it will be a native lean causal inference model rather than a bloated multimodal model,

Why not both. Gemma 3 is native multimodal, but you don't have to download the mmproj adapter, so there's no bloat for those who don't care about vision. And more importantly, pretraining on images allows the model to get more information about the world, enhancing its general performance.
There's literally no downsides, only upsides.