New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fsdp inference checkpoints #39
Conversation
inference/inference.py
Outdated
parent_directory = os.path.dirname(current_directory) | ||
|
||
# Append the parent directory to sys.path | ||
sys.path.append(parent_directory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please move the code for path setting inside main / function
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@HamidShojanazeri Thanks for getting this started. Please see comments inline
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add steps used for testing this out, including below along with output of your test for verification:
- command to run and generate the pure fsdp checkpoint
- command for converting the checkpoint to HF format for inference
- command for running the inference
done. |
Adding converter script for converting FSDP sharded state dict to HF model for inference.