New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decouple MLflow Components #1549

Closed

fa9r wants to merge 21 commits into develop from feature/decouple-mlflow-components

Contributor

fa9r commented May 22, 2023 •

edited

Loading

Describe changes

This PR decouples the MLflow deployer and model registry from the experiment tracker so they can be used without any models having been logged beforehand.

Example

@step
def digits_data_loader():
    ...

@step  # No experiment tracker used here
def svc_trainer(X_train: np.ndarray, y_train: np.ndarray) -> ClassifierMixin:
    model = SVC(gamma=0.001)
    model.fit(X_train, y_train)
    return model

from zenml.integrations.mlflow.steps.mlflow_deployer import mlflow_model_deployer_step

@pipeline
def first_pipeline():
    X_train, X_test, y_train, y_test = digits_data_loader()
    model = svc_trainer(X_train, y_train)
    mlflow_model_deployer_step(model)

Implementation Details

Both mlflow_model_deployer_step and mlflow_register_model_step now log the model to MLflow in a new run if the model was not found; both steps were also heavily refactored
Instead of getting the MLflow connection from the experiment tracker in the stack, the model registry and deployer can now be configured separately
To achieve this, MLFlowStackComponentMixin and MLFlowConfigMixin classes were added which all three MLflow components now inherit from

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
If I have added an integration, I have updated the integrations table and the corresponding website section.
I have added tests to cover my changes.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

fa9r added 5 commits

May 22, 2023 10:16


          Remove mlflow_utils.get_tracking_uri()

385f05f


          Move MLFlowExperimentTracker._local_mlflow_backend() to mlflow_utils.py


          Move mlflow connection config args to new MLFlowConfigMixin class


          Move core mlflow client logic to new MLFlowStackComponentMixin class

3f9cd82


          Decouple MLflow deployer and registry from experiment tracker

e1dc238

github-actions bot added internal enhancement labels

fa9r added 5 commits

May 22, 2023 14:29


          Fix tests

63e8b3d


          Cleanup

a17fb4c


          Merge branch 'develop' into feature/decouple-mlflow-components

8d217b2


          Adjust example post execution to fetch tracking URI from metadata

d7e0808


          Move all mlflow calls from mlflow_utils to MLFlowStackComponentMixin

65ebca4

review-notebook-app bot commented May 31, 2023

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

fa9r added 9 commits

May 31, 2023 11:50


          Cleanup

b6c3581


          🚀 MLflow deployment without experiment tracker works 🚀

d0205d7


          Clean mlflow_model_deployer_step and make it backwards compatible

588c0d5


          Move find_or_log_model > MLflowStackComponentMixin.find_or_log_mlflow…

2b9f04a

…_model


          Don't log models into nested runs


          Adjust mlflow_register_model_step to work without experiment tracker

87972e3


          Clean up MLflow examples

f7847e8


          Adjust docs

480ff50


          Merge branch 'develop' into feature/decouple-mlflow-components

fbcacf1

fa9r marked this pull request as ready for review

May 31, 2023 17:53

fa9r requested a review from safoinme

May 31, 2023 18:00

fa9r changed the title ~~WIP: Decouple MLflow components~~ Decouple MLflow Components

fa9r requested a review from stefannica

May 31, 2023 18:01


          Fix integration tests

d5b8aa8

Contributor

github-actions bot commented Jun 2, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

2 similar comments

Contributor

github-actions bot commented Jun 2, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 5, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.


          Fix MLflow tracking integration test

8b78c54

Contributor

github-actions bot commented Jun 5, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

20 similar comments

Contributor

github-actions bot commented Jun 6, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 6, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 7, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 7, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 8, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 8, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 9, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 9, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 12, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 12, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 13, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 13, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 14, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 14, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 15, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 15, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 16, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 16, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 19, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

Contributor

github-actions bot commented Jun 19, 2023

@stefannica, @safoinme
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

safoinme mentioned this pull request

bump MLflow to 2.5.0 #1708

Merged

8 tasks

fa9r marked this pull request as draft

November 30, 2023 09:21

fa9r removed request for stefannica and safoinme

November 30, 2023 09:21

CLAassistant commented Jan 30, 2024 •

edited

Loading

All committers have signed the CLA.

htahir1 closed this

Contributor

htahir1 commented Jul 9, 2024

branch was too stale, we'll revisit if we come back to this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement internal