Hello, Apple Team!

[win10ogod]

@summmeer

I'm a member of the open-source community currently working on a project focused on converting autoregressive language models into diffusion-based models like Dream. We understand that direct code contributions may raise legal or compliance concerns. However, we are wondering if any experienced members of your team would be open to providing informal guidance or acting as advisors on this effort.

Even limited technical insights or feedback—shared in a personal or non-official capacity—would be incredibly valuable to help the broader community explore this new model architecture direction responsibly and effectively.

We truly appreciate any support or consideration you can offer.

Best regards,
win10ogod.

[win10ogod]

You can contact me, my email:
jmes100010@gmail.com

[dreasysnail]

Hi @win10ogod, thanks for reaching out. We’re exploring ways to enable community contributions. Please stay tuned. In the meantime, feel free to fork our repo and experiment. If you develop anything you think could benefit the repo, let us know – we may be able to integrate it.

[win10ogod]

Hi @win10ogod, thanks for reaching out. We’re exploring ways to enable community contributions. Please stay tuned. In the meantime, feel free to fork our repo and experiment. If you develop anything you think could benefit the repo, let us know – we may be able to integrate it.

We want to know how to convert autoregressive models such as qwen2.5 into Dream, because there are some technical barriers (? I don’t know if it is true), but the community does not have a successful model conversion case

[Pr0fe5s0r]

@dreasysnail Hey, Just wondering. How much GPU hours it took to train a diffucoder model? I think it is same as DiffuLLama right?

[dreasysnail]

Hi @win10ogod, thanks for reaching out. We’re exploring ways to enable community contributions. Please stay tuned. In the meantime, feel free to fork our repo and experiment. If you develop anything you think could benefit the repo, let us know – we may be able to integrate it.

We want to know how to convert autoregressive models such as qwen2.5 into Dream, because there are some technical barriers (? I don’t know if it is true), but the community does not have a successful model conversion case

We are following DiffuLLaMA's approach to convert QWEN models to diffusion models

[dreasysnail]

@dreasysnail Hey, Just wondering. How much GPU hours it took to train a diffucoder model? I think it is same as DiffuLLama right?

Yes it's using similar recipe to DiffuLLaMA. Roughly 3k GPU hours on 65B data.

[summmeer]

@Pr0fe5s0r If you're using H100, the total wall-clock time for training on 65B tokens was approximately 40 hours on 10 nodes (that is to say 3200 H100 hours)