Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for automatically registering models to UC at the end of training #618

Merged
merged 67 commits into from
Oct 17, 2023

Conversation

dakinggg
Copy link
Collaborator

@dakinggg dakinggg commented Sep 21, 2023

Adds support for directly logging models to mlflow during training for easy deployment on databricks model serving. Tested and was able to deploy both an MPT model and a llama model succesfully.

  • Confirm can log MPT and llama and deploy
  • Measure time of final checkpoint for different sizes
  • wait for next composer patch release with mlflow model logging changes
  • write tests
  • add support for using UC
  • refactor to skip the extra load to memory.

along for the ride: fixing the hf checkpointer to work with non fsdp models as well

  • redo manual tests once ready for review and unit tests pass (successfully logged and deployed mpt and llama models)

@dakinggg dakinggg marked this pull request as ready for review September 27, 2023 01:39
Copy link
Collaborator

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What scale of models can this take? What happens if I try shoving in a 30b?

Should this be here instead of the mlflow logger as a public fn that this callback calls?

@dakinggg
Copy link
Collaborator Author

@mvpatel2000 large models work. at least 30b, haven't had cluster space to run 70b yet. this does call the mlflow logger log_model from composer. the work in this callback is just preparing the inputs to that function call.

llmfoundry/callbacks/hf_checkpointer.py Show resolved Hide resolved
llmfoundry/callbacks/hf_checkpointer.py Outdated Show resolved Hide resolved
llmfoundry/callbacks/hf_checkpointer.py Show resolved Hide resolved
llmfoundry/callbacks/hf_checkpointer.py Show resolved Hide resolved
@dakinggg dakinggg marked this pull request as ready for review October 12, 2023 01:23
@dakinggg dakinggg changed the title Mlflow log model Add support for automatically registering models to UC at the end of training Oct 13, 2023
Copy link
Contributor

@eracah eracah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok lgtm

@dakinggg dakinggg merged commit cc238a3 into mosaicml:main Oct 17, 2023
12 checks passed
@dakinggg dakinggg deleted the mlflow-log-model branch October 17, 2023 04:39
jjanezhang added a commit that referenced this pull request Oct 20, 2023
fixed typing for mosaicmllogger

fixed formatting, moved logging logic out of logger builder

removed callback moved all to train.py

Convert to DataSpec and add token counts that include padding (#676)

Add support for automatically registering models to UC at the end of training (#618)

removed callback from init

removed callback from init

removed generate callback

sorted imports

formatting changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants