Support Interleaved Multimodal Sequences and Long-Context Training Infrastructure

[Sishxo]

Hi, thanks for the great work on Cheers! I would like to suggest adding support for fully interleaved multimodal sequences covering both input and output, where text and images can appear alternately (e.g., image → text → image → text) in both prompt and generated responses. This is important for realistic multimodal scenarios such as multi-image reasoning, document-style understanding, and unified multimodal modeling, but current pipelines are mostly limited to single-image input and text-only generation. Supporting this would require unified formatting/tokenization, modality-aware masking for training (loss and attention), and consistent handling in both training and inference. This feature would significantly improve Cheers’ ability to model richer multimodal interactions and long-context multimodal sequences.

[PengDa02]

This is a very interesting and important suggestion—thanks for bringing it up.

We fully agree that supporting fully interleaved multimodal sequences (for both inputs and outputs) is crucial, especially for long-context scenarios and richer multimodal interactions such as multi-image reasoning and document-style understanding. It’s a natural next step toward more unified multimodal modeling. In fact, our current framework already provides preliminary support for interleaved image-text understanding on the input side. We are actively working on extending this toward more complete and consistent support.

We also welcome contributions from the community—if you’re interested in exploring this direction, feel free to modify the framework and open a PR.