-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Megatron-VLM training #806
base: main
Are you sure you want to change the base?
Conversation
Hi. Thanks for creating this PR. We (NVIDIA) are actually planning to release VLM training functionality in Megatron core in the next couple of weeks. As you may have seen, we've been pushing out some preparatory code to support this. Our initial example release is going to be pretraining and SFT for a llava architecture model using llama3 and clip backbones and a general multimodal webdataset based dataloader. We're reviewing your PR internally to see if we can incorporate any of your work alongside ours and will be sure to credit you as such if we do. Thanks again! |
Thank you for your attention! Looking forward to the official implementation! |
Hello, i have a question about this PR: how will vit and llm split in PP stage with independent_parallel = True? Thank you! |
@wangxiang2713 ViT will be in the first stage of LM. |
Me fale mais suas dúvidas
Em qui, 13 de jun de 2024 05:48, Qingsong Lv ***@***.***>
escreveu:
… @wangxiang2713 <https://github.com/wangxiang2713> ViT will be in the
first stage of LM.
—
Reply to this email directly, view it on GitHub
<#806 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BH4G62IYR7X62XTHLIL7OUTZHFMG3AVCNFSM6AAAAABHHYOBIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVGAZDSMJSHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
In this pull request, we open source our solution for visual-language model training and inference in pure Megatron style code. In this codebase, we support:
The running example is in
examples/llava
folder.Hope that our work can contribute to the open source community. If there are any questions, welcome feedback!