Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ENHANCEMENT] Do you have a plan that supports Mixtral 8x7B? #649

Open
matrixssy opened this issue Jan 4, 2024 · 7 comments
Open

[ENHANCEMENT] Do you have a plan that supports Mixtral 8x7B? #649

matrixssy opened this issue Jan 4, 2024 · 7 comments
Labels
stale No activity in 60 days on issue or PR

Comments

@matrixssy
Copy link

matrixssy commented Jan 4, 2024 •
edited

No description provided.

@cdj0311
Copy link

cdj0311 commented Jan 14, 2024

The same question.

@matrixssy
Copy link
Author

The same question.

I am working for it, but I am not sure if it will be accepted.

@bityigoss
Copy link

@matrixssy any progress ? thank you.

@matrixssy matrixssy mentioned this issue Jan 18, 2024
Open
@matrixssy
Copy link
Author

see #667

@yanring
Copy link
Contributor

yanring commented Feb 15, 2024

Hi, Please refer to this script for MoE/Mixtral training.

@matrixssy
Copy link
Author

matrixssy commented Feb 17, 2024 •
edited

Hi, Please refer to this script for MoE/Mixtral training.

Great! But I would like to know how to convert Hugging Face (HF) weights to Megatron (MG) format, and if it's possible to convert them back after training?

@github-actions GitHub Actions
Copy link

Marking as stale. No activity in 60 days.

@github-actions github-actions bot added the stale No activity in 60 days on issue or PR label Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale No activity in 60 days on issue or PR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cdj0311 @bityigoss @yanring @matrixssy