Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add v3 converter #10

Merged
merged 19 commits into from
Aug 14, 2024
Merged

Add v3 converter #10

merged 19 commits into from
Aug 14, 2024

Conversation

YumaTsuta
Copy link
Collaborator

v3 model用 megatron -> huggingface コンバーターを追加

@YumaTsuta

This comment was marked as outdated.

YumaTsuta and others added 3 commits August 7, 2024 19:27
deal wirh relative path setting on source checkpoint
@YumaTsuta YumaTsuta marked this pull request as ready for review August 8, 2024 07:46
@YumaTsuta YumaTsuta self-assigned this Aug 8, 2024
@YumaTsuta
Copy link
Collaborator Author

YumaTsuta commented Aug 8, 2024

#12 への対応前提のため、動作確認時には environment/src/llm-jp-tokenizer で v3.0b2 ブランチに事前に切り替えてください

git fetch
git checkout v3.0b2

@YumaTsuta
Copy link
Collaborator Author

#12 merge完了

pretrain/scripts/v3-converter/convert.sh Outdated Show resolved Hide resolved
pretrain/scripts/v3-converter/convert.sh Outdated Show resolved Hide resolved
@YumaTsuta

This comment was marked as resolved.

@YumaTsuta YumaTsuta requested a review from odashi August 13, 2024 15:08
pretrain/scripts/v3-converter/convert.sh Outdated Show resolved Hide resolved
pretrain/scripts/v3-converter/convert.sh Outdated Show resolved Hide resolved
YumaTsuta and others added 2 commits August 14, 2024 08:44
Co-authored-by: Yusuke Oda <yusuke.oda@predicate.jp>
Co-authored-by: Yusuke Oda <yusuke.oda@predicate.jp>
@YumaTsuta
Copy link
Collaborator Author

1ノード占有は勿体無い気もしますが処理時間が短い(13Bで5分以内, 172Bで約3時間)ので問題ないかと思いsuggestionに変更しました

@YumaTsuta YumaTsuta requested a review from odashi August 13, 2024 23:55
@YumaTsuta YumaTsuta merged commit d2866fd into main Aug 14, 2024
@YumaTsuta YumaTsuta deleted the add-v3-converter branch August 14, 2024 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants