AutoML support for MLflow Pipelines using FLAML #6959

mshtelma · 2022-10-03T08:42:50Z

Signed-off-by: Michael Shtelma michael.shtelma@databricks.com

Related Issues/PRs

What changes are proposed in this pull request?

This PR adds AutoML support to MLflow Pipelines using FLAML library.

How is this patch tested?

I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly by following the steps below.

Click the Details link on the Preview docs check.
Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

Support training via AutoML using FLAML in MLflow Pipelines.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Signed-off-by: mshtelma <mshtelma@gmail.com>

Signed-off-by: Prithvi Kannan <prithvi.kannan@databricks.com>

# Conflicts: # mlflow/pipelines/steps/train.py

mlflow-automation · 2022-10-10T11:05:42Z

Documentation preview will be available here.

Notes

Ignore this comment if this PR does not change the documentation.
It takes a few minutes for the preview to be available.
The preview is updated on every commit to this PR.
Job URL: https://circleci.com/gh/mlflow/mlflow/33203
Updated at: 2022-10-14 21:49:00.554120

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

prithvikannan

Thanks for building this out @mshtelma -- this is awesome! I've left a few small comments, mostly about code styling.

prithvikannan · 2022-10-10T22:05:20Z

mlflow/pipelines/steps/automl/flaml.py

+    # pylint: disable=keyword-arg-before-vararg
+    # pylint: disable=unused-argument


Why do we need to do this? Can we just remove labels, weight_val, and weight_train and these pylint lines?

This is the signature that FLAML expects, so I believe we should follow it.

prithvikannan · 2022-10-10T22:08:44Z

mlflow/pipelines/steps/automl/flaml.py

+        else:
+            _logger.warning(
+                f"There is no FLAML alternative or custom metric for {primary_metric} metric.\n"
+                f"Using 'auto' metric instead."
+            )
+            metric = "auto"


I think that we should throw an error in this case. We should require the user to specify a primary metric (which we do), and either respect that metric or error.

prithvikannan · 2022-10-10T22:12:08Z

mlflow/pipelines/steps/automl/flaml.py

+        automl_settings["metric"] = metric
+        automl_settings["task"] = "regression"
+
+        mlflow.autolog(disable=True)


Why do we have to disable autologging here? In microsoft/FLAML#747, we fixed the mlflow-FLAML integration so that the logging is in the form we want.

If I comment out this line, I get a weird exception in our auto-logging code. Btw, it looks like an MLflow bug:

Traceback (most recent call last): File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/pipelines/steps/automl/flaml.py", line 114, in _create_model_automl automl.fit(X, y, **automl_settings) File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/automl.py", line 2898, in fit self._search() File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/automl.py", line 3462, in _search self._search_sequential() File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/automl.py", line 3280, in _search_sequential analysis = tune.run( File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/tune/tune.py", line 514, in run result = evaluation_function(trial_to_run.config) File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/automl.py", line 350, in _compute_with_config_base ) = compute_estimator( File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/ml.py", line 610, in compute_estimator val_loss, metric_for_logging, train_time, pred_time = evaluate_model_CV( File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/ml.py", line 523, in evaluate_model_CV val_loss_i, metric_i, train_time_i, pred_time_i = get_val_loss( File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/ml.py", line 412, in get_val_loss estimator.fit(X_train, y_train, budget, **fit_kwargs) File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/model.py", line 1133, in fit self._fit( File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/flaml/model.py", line 197, in _fit model.fit(X_train, y_train, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 555, in safe_patch_function patch_function(call_original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 254, in patch_with_managed_run result = patch_function(original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/sklearn/__init__.py", line 1552, in patched_fit result = fit_impl(original, self, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/sklearn/__init__.py", line 1299, in fit_mlflow_xgboost_and_lightgbm fit_output = original(self, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 536, in call_original return call_original_fn_with_event_logging(_original_fn, og_args, og_kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 471, in call_original_fn_with_event_logging original_fn_result = original_fn(*og_args, **og_kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 533, in _original_fn original_result = original(*_og_args, **_og_kwargs) File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/lightgbm/sklearn.py", line 895, in fit super().fit(X, y, sample_weight=sample_weight, init_score=init_score, File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 555, in safe_patch_function patch_function(call_original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 254, in patch_with_managed_run result = patch_function(original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/sklearn/__init__.py", line 1559, in patched_fit return original(self, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 536, in call_original return call_original_fn_with_event_logging(_original_fn, og_args, og_kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 471, in call_original_fn_with_event_logging original_fn_result = original_fn(*og_args, **og_kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 533, in _original_fn original_result = original(*_og_args, **_og_kwargs) File "/opt/homebrew/Caskroom/miniconda/base/envs/mlflow-dev-env/lib/python3.8/site-packages/lightgbm/sklearn.py", line 748, in fit self._Booster = train( File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 555, in safe_patch_function patch_function(call_original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/safety.py", line 254, in patch_with_managed_run result = patch_function(original, *args, **kwargs) File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/lightgbm.py", line 665, in train param_logging_operations.await_completion() File "/Users/michael.shtelma/Development/Projects/mlflow_msh/mlflow/utils/autologging_utils/client.py", line 69, in await_completion raise MlflowException( mlflow.exceptions.MlflowException: The following failures occurred while performing one or more logging operations: [MlflowException('Failed to perform one or more operations on the run with ID 91aa473ed5cd417184e4de742889c273. Failed operations: [MlflowException(\'Changing param values is not allowed. Params were already logged=\\\'[{\\\'key\\\': \\\'boosting_type\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'gbdt\\\'}, {\\\'key\\\': \\\'max_depth\\\', \\\'old_value\\\': \\\'0\\\', \\\'new_value\\\': \\\'-1\\\'}, {\\\'key\\\': \\\'min_child_samples\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'20\\\'}, {\\\'key\\\': \\\'min_child_weight\\\', \\\'old_value\\\': \\\'0.9999999999999993\\\', \\\'new_value\\\': \\\'0.001\\\'}, {\\\'key\\\': \\\'min_split_gain\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'0.0\\\'}, {\\\'key\\\': \\\'num_leaves\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'4\\\'}, {\\\'key\\\': \\\'subsample_for_bin\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'200000\\\'}, {\\\'key\\\': \\\'subsample_freq\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'0\\\'}, {\\\'key\\\': \\\'max_bin\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'255\\\'}, {\\\'key\\\': \\\'verbose\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'-1\\\'}, {\\\'key\\\': \\\'objective\\\', \\\'old_value\\\': \\\'reg:squarederror\\\', \\\'new_value\\\': \\\'regression\\\'}, {\\\'key\\\': \\\'metric\\\', \\\'old_value\\\': None, \\\'new_value\\\': "[\\\'regression\\\']"}, {\\\'key\\\': \\\'feature_name\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'auto\\\'}, {\\\'key\\\': \\\'categorical_feature\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'auto\\\'}, {\\\'key\\\': \\\'evals_result\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'None\\\'}, {\\\'key\\\': \\\'verbose_eval\\\', \\\'old_value\\\': \\\'True\\\', \\\'new_value\\\': \\\'warn\\\'}, {\\\'key\\\': \\\'keep_training_booster\\\', \\\'old_value\\\': None, \\\'new_value\\\': \\\'False\\\'}]\\\' for run ID=\\\'91aa473ed5cd417184e4de742889c273\\\'.\')]')]

Mind adding a ticket to follow up on this issue?

https://databricks.atlassian.net/browse/ML-26033

prithvikannan · 2022-10-10T22:12:45Z

mlflow/pipelines/steps/automl/flaml.py

+        _logger.warning(
+            f"Error has occurred during training of AutoML model using FLAML: {repr(e)}",
+            exc_info=True,
+        )


I don't think we need this logging since we will throw the exception with this text.

prithvikannan · 2022-10-10T22:15:49Z

mlflow/pipelines/steps/train.py

+        if "estimator_method" not in self.step_config and self.step_config["using"] in [
+            "estimator_spec"
+        ]:
            raise MlflowException(
                "Missing 'estimator_method' configuration in the train step.",
                error_code=INVALID_PARAMETER_VALUE,
            )


Can we update the exception message to mention using estimator_spec?

prithvikannan · 2022-10-10T22:21:24Z

mlflow/pipelines/steps/train.py

+        return estimator
+
+    def _resolve_estimator_plugin(self, plugin_str, X_train, y_train):
+        plugin_str = plugin_str.replace("/", ".").replace("@", ".")


Why do we need this? Can we just make the user plugin str automl.flaml?

The idea is to have some sort of a plugin mechanism so that in the future, if we decide to add AutoGluon, for example, we can just implent autogluon.py, and "automl/autogluon" will work out of the box.

prithvikannan · 2022-10-10T22:22:53Z

mlflow/pipelines/steps/train.py

+        plugin_module_str = f"{sys.modules[__name__].__package__}.{plugin_str}"
+        estimator_fn = getattr(importlib.import_module(plugin_module_str), "get_estimator")


Can we follow the pattern on L373-374 instead?

prithvikannan · 2022-10-10T22:25:19Z

mlflow/pipelines/steps/automl/flaml.py

+}
+
+
+def get_estimator(


Can we rename this to get_estimator_and_best_params?

prithvikannan · 2022-10-10T22:30:36Z

mlflow/pipelines/utils/metrics.py

+def _load_one_custom_metric_function(pipeline_root: str, metric: PipelineMetric) -> callable:
+    try:
+        sys.path.append(pipeline_root)
+        custom_metrics_mod = importlib.import_module("steps.custom_metrics")
+        return getattr(custom_metrics_mod, metric.custom_function)
+    except Exception as e:
+        raise MlflowException(
+            message="Failed to load custom metric functions",
+            error_code=BAD_REQUEST,
+        ) from e
+
+


Can reuse the existing _load_custom_metric_functions by just specifying a single metric in the metrics list?

Great point. Done!

jinzhang21

Reviewed the first 2 files. Will take a look at the rest later. Thanks, @mshtelma !

jinzhang21 · 2022-10-11T15:37:10Z

mlflow/pipelines/steps/automl/flaml.py

+AUTOML_DEFAULT_TIME_BUDGET = 10
+MLFLOW_TO_FLAML_METRICS = {


Add "_" in front since we don't want to expose these params externally.
_AUTOML_DEFAULT_TIME_BUDGET and _MLFLOW_TO_FLAML_METRICS

jinzhang21 · 2022-10-11T15:40:16Z

mlflow/pipelines/steps/automl/flaml.py

+            "time_budget_secs", AUTOML_DEFAULT_TIME_BUDGET
+        )
+        automl_settings["metric"] = metric
+        automl_settings["task"] = "regression"


Can we pull automl_settings["task"] as the function arg? We are building the classification pipeline right now. It'd make sense to support both problem types in the train step.

Great point! Done!

jinzhang21 · 2022-10-11T15:41:53Z

mlflow/pipelines/steps/automl/flaml.py

+        _logger.warning(
+            f"Error has occurred during training of AutoML model using FLAML: {repr(e)}",
+            exc_info=True,
+        )


I don't think it's necessary to both log and raise. Simple raise should suffice.

jinzhang21 · 2022-10-11T15:43:59Z

mlflow/pipelines/steps/automl/flaml.py

+        mlflow.autolog(disable=True)
+        automl = AutoML()
+        automl.fit(X, y, **automl_settings)
+        mlflow.autolog(disable=False, log_models=False)


Can we add a comment on the expected autolog behavior change here?

Done. It's looking like a bug in autologging functionality.

# Conflicts: # mlflow/pipelines/steps/train.py

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

jinzhang21 · 2022-10-14T02:57:05Z

mlflow/pipelines/steps/automl/flaml.py

+        automl_settings["metric"] = metric
+        automl_settings["task"] = "regression"
+
+        mlflow.autolog(disable=True)


Mind adding a ticket to follow up on this issue?

jinzhang21 · 2022-10-14T02:59:05Z

mlflow/pipelines/steps/train.py

@@ -135,14 +136,15 @@ def _validate_and_apply_step_config(self):
                error_code=INVALID_PARAMETER_VALUE,
            )
        self.skip_data_profiling = self.step_config.get("skip_data_profiling", False)
-        if "estimator_method" not in self.step_config:
+        if "estimator_method" not in self.step_config and self.step_config["using"] in [


self.step_config["using"] == "estimator_spec"? The exception message below is specific to this condition.

Yes, exactly.

jinzhang21 · 2022-10-14T03:03:32Z

mlflow/pipelines/steps/train.py

+                    f"<pre>{param}: {value}</pre><br>"
+                    for param, value in self.best_parameters.items()


Can user simply copy these params to pipeline.yaml to create a baseline model?

Yes, of course, this should work.

jinzhang21 · 2022-10-14T03:07:25Z

mlflow/pipelines/steps/train.py

@@ -887,7 +946,7 @@ def objective(X_train, y_train, validation_df, hyperparameter_args, on_worker=Fa
    def _log_estimator_to_mlflow(self, estimator, X_train_sampled, on_worker=False):
        from mlflow.models.signature import infer_signature

-        if hasattr(estimator, "best_score_"):
+        if hasattr(estimator, "best_score_") and (type(estimator.best_score_) in [int, float]):


What other types could best_score_ be? None?

The issue is that in some cases (lgbm as far as I remember) it can be a Dict. It is not related to AutoML, I have seen this error before as well, with AutoML it's just easier to reproduce it.

jinzhang21 · 2022-10-14T03:14:38Z

tests/pipelines/test_train_step.py

@@ -60,7 +60,14 @@ def setup_train_dataset(pipeline_root: Path):


 # Sets up the constructed TrainStep instance
-def setup_train_step(pipeline_root: Path, use_tuning: bool, with_hardcoded_params: bool = True):
+def setup_train_step(


Can we split this function into setup_train_step_with_tuning and setup_train_step_with_automl? They are not very related. The function has become difficult to reasonable with so many knobs.

jinzhang21

Thanks, @mshtelma ! LGTM overall with some minor comments. Please attach a screenshot on the step card output. Thanks.

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

mshtelma · 2022-10-14T18:24:15Z

Here is the screenshot of the train step card:

prithvikannan

LGTM, thanks @mshtelma for driving this effort!

prithvikannan · 2022-10-14T18:27:42Z

mlflow/pipelines/steps/train.py

@@ -61,8 +61,9 @@ def __init__(self, step_config, pipeline_root, pipeline_config=None):
        self.pipeline_config = pipeline_config

    def _validate_and_apply_step_config(self):
+        self.task = self.step_config.get("template_name", "regression/v1").rsplit("/", 1)[0]


nice - clever way to not hardcode the flaml task

prithvikannan

Sorry, one small comment about writing outputs to yaml

prithvikannan · 2022-10-14T18:33:37Z

mlflow/pipelines/steps/train.py

+        if self.step_config["using"].startswith("automl"):
+            automl_card_tab = card.add_tab(
+                "Best Estimator (FLAML)",
+                "{{ AUTOML }} ",
+            )
+            params_html = "".join(
+                [f"<pre>{param}: {value}</pre>" for param, value in self.best_parameters.items()]
+            )
+            automl_card_tab.add_html(
+                "AUTOML",
+                f"<b>Best estimator:</b><br>"
+                f"<pre>{self.best_estimator_name}</pre><br>"
+                f"<b>Best estimator class:</b><br>"
+                f"<pre>{self.best_estimator_class}</pre><br><br>"
+                f"<b>Best parameters:</b><br>"
+                f"{params_html}<br><br>",
+            )
+


can we write the automl parameters output to a yaml file? and also log that file to mlflow? this is important for getting users from automl -> estimator spec training.

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

prithvikannan

Thanks for this change! Left a small comment about changing the best_parameters.yaml syntax, but we are almost there

prithvikannan · 2022-10-14T20:12:29Z

mlflow/pipelines/steps/train.py

@@ -414,6 +414,7 @@ def _resolve_estimator_plugin(self, plugin_str, X_train, y_train):
            f"{estimator.__class__.__module__}.{estimator.__class__.__name__}"
        )
        self.best_parameters = best_parameters
+        self._write_best_parameters_outputs(best_parameters, {}, output_directory)


I think we will need to change the behavior in _write_best_parameters_outputs.

Currently it will output in this form

# tuned hyperparameters alpha: 0.1 # hardcoded parameters

but that doesn't make sense in the context of automl.

Do we need to write an empty "# hardcoded parameters" bucket for tuning runs? I will check if the respective dict has any values and write the header only in this case. Hope this is ok for you.

sorry my comment was unclear. the issue is that automl parameters will be written under the heading "# tuned hyperparameters", but that is not accurate. instead, we should extend _write_best_parameters_outputs to accept a automl parameters argument, and then output those under the heading "# automl parameters"

the empty dict for hardcoded params is not a problem IMO

ok, pushed this one as well!

prithvikannan · 2022-10-14T20:14:58Z

mlflow/pipelines/steps/train.py

@@ -642,6 +681,25 @@ def render_schema(inputs, title):
                f"<b>Best parameters:</b><br>" f"<pre>{best_parameters}</pre><br><br>",
            )

+        # Tab 8.1: Best AutoML Parameters
+        if self.step_config["using"].startswith("automl"):


i think we can collapse this into the if os.path.exists(best_parameters_yaml): section above now that we are also writing that file

ok, for AutoML, we also output the name and class of the estimator.

…parameters yaml Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

prithvikannan

LGTM! Thanks for working through many iterations of this @mshtelma

…parameters yaml Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

ML-25576 AutoML support for MLflow Pipelines using FLAML

8f802d3

Signed-off-by: mshtelma <mshtelma@gmail.com>

dbczumar changed the title ~~ML-25576 AutoML support for MLflow Pipelines using FLAML~~ AutoML support for MLflow Pipelines using FLAML Oct 3, 2022

adding flaml to test requirements

b8ecc2e

Signed-off-by: Prithvi Kannan <prithvi.kannan@databricks.com>

github-actions bot added the area/recipes MLflow Recipes, Recipes APIs, Recipes configs, Recipe Templates label Oct 3, 2022

prithvikannan added the rn/feature Mention under Features in Changelogs. label Oct 3, 2022

prithvikannan and others added 5 commits October 3, 2022 13:17

merge issue

e1dfdb9

Signed-off-by: Prithvi Kannan <prithvi.kannan@databricks.com>

remove import helper

0a2872f

Signed-off-by: Prithvi Kannan <prithvi.kannan@databricks.com>

dont tell users to add to requirements.txt

7234f5f

Signed-off-by: Prithvi Kannan <prithvi.kannan@databricks.com>

Merge branch 'master' into mlp_automl_support

9b6b089

# Conflicts: # mlflow/pipelines/steps/train.py

AutoML Support for MLP - FLAML Integration

8160dd6

mshtelma added 2 commits October 10, 2022 14:46

AutoML Support for MLP - FLAML Integration

fd9d298

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

formatting

fd9e516

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

mshtelma requested review from jinzhang21 and prithvikannan October 10, 2022 16:07

prithvikannan reviewed Oct 10, 2022

View reviewed changes

jinzhang21 reviewed Oct 11, 2022

View reviewed changes

mshtelma and others added 4 commits October 12, 2022 21:43

Merge branch 'mlflow:master' into mlp_automl_support

6cbdb3c

Merge branch 'master' into mlp_automl_support

7de5714

# Conflicts: # mlflow/pipelines/steps/train.py

Merge

97d974a

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

Small refactoring

d3b4a7b

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

jinzhang21 reviewed Oct 14, 2022

View reviewed changes

mshtelma and others added 3 commits October 14, 2022 08:26

Merge branch 'mlflow:master' into mlp_automl_support

9046cf5

Small refactoring

d89ffeb

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

Small refactoring

5cf565f

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

jinzhang21 approved these changes Oct 14, 2022

View reviewed changes

mshtelma added 3 commits October 14, 2022 19:21

Small refactoring

c08d2a6

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

Small refactoring

de17519

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

Formatting

27541e5

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

prithvikannan approved these changes Oct 14, 2022

View reviewed changes

prithvikannan requested changes Oct 14, 2022

View reviewed changes

Wrting best prameters to yaml and also logging it to mlflow

4e737ad

Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

mshtelma requested a review from prithvikannan October 14, 2022 19:12

prithvikannan requested changes Oct 14, 2022

View reviewed changes

Refactoring around Card for AutoMKL/Tuning and cosmetics around best …

faacbea

…parameters yaml Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

mshtelma requested a review from prithvikannan October 14, 2022 20:57

Refactoring around Card for AutoMKL/Tuning and cosmetics around best …

e2f9e86

…parameters yaml Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

prithvikannan approved these changes Oct 14, 2022

View reviewed changes

Refactoring around Card for AutoMKL/Tuning and cosmetics around best …

abc7a4b

…parameters yaml Signed-off-by: Michael Shtelma <michael.shtelma@databricks.com>

mshtelma merged commit f947f73 into mlflow:master Oct 14, 2022

prithvikannan mentioned this pull request Nov 12, 2022

FLAML classifier metrics #7322

Merged

33 tasks

		# pylint: disable=keyword-arg-before-vararg
		# pylint: disable=unused-argument

		plugin_module_str = f"{sys.modules[__name__].__package__}.{plugin_str}"
		estimator_fn = getattr(importlib.import_module(plugin_module_str), "get_estimator")

		f"<pre>{param}: {value}</pre><br>"
		for param, value in self.best_parameters.items()

		AUTOML_DEFAULT_TIME_BUDGET = 10
		MLFLOW_TO_FLAML_METRICS = {

AutoML support for MLflow Pipelines using FLAML #6959

AutoML support for MLflow Pipelines using FLAML #6959

Conversation

mshtelma commented Oct 3, 2022 • edited by prithvikannan Loading

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

mlflow-automation commented Oct 10, 2022 • edited Loading

Documentation preview will be available here.

prithvikannan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinzhang21 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinzhang21 left a comment

Choose a reason for hiding this comment

mshtelma commented Oct 14, 2022

prithvikannan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prithvikannan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prithvikannan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prithvikannan left a comment

Choose a reason for hiding this comment

mshtelma commented Oct 3, 2022 •

edited by prithvikannan

Loading

mlflow-automation commented Oct 10, 2022 •

edited

Loading