Fix autologging compatibility with Keras >= 2.6.0 #4766

dbczumar · 2021-09-03T04:20:52Z

What changes are proposed in this pull request?

Fix autologging compatibility with Keras >= 2.6.0, addressing the following issues:

Calling mlflow.autolog() followed by import tensorflow set up monkey patches for TensorFlow and Keras autologging at the same time, causing autologging to fail and no run content to be created
Calling mlflow.autolog() set up an import hook that erroneously mutated references to tensorflow.keras.Model, causing them to point to a deprecated class
Calling mlflow.autolog() followed by import keras attempted to enable tensorflow autologging before tf.keras was available, causing enablement to fail and a warning to be printed

How is this patch tested?

Included unit tests

Release Notes

Fix autologging compatibility bugs with TensorFlow and Keras versions >= 2.6.0.

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

github-actions · 2021-09-03T04:21:07Z

@dbczumar Thanks for the contribution! The DCO check failed. Please sign off your commits by following the instructions here: https://github.com/mlflow/mlflow/runs/3502680818. See https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst#sign-your-work for more details.

dbczumar · 2021-09-03T04:24:25Z

mlflow/tracking/fluent.py

+        if autologging_is_disabled("tensorflow"):
+            setup_autologging(tensorflow_module)


By design, importing keras enables tensorflow autologging. We insert this check to ensure that importing tensorflow, which imports keras, does not enable tensorflow autologging twice.

Keras needs to enable tensorflow autologging so that import keras enables tensorflow autologging and import tensorflow only enables tensorflow autologging and not keras autologging.

dbczumar · 2021-09-03T04:25:10Z

mlflow/tracking/fluent.py

+            # autologging in Keras >= 2.6.0
+            try:
+                import tensorflow
+                setup_autologging(tensorflow)


Keras needs to enable tensorflow autologging so that import keras enables tensorflow autologging and import tensorflow only enables tensorflow autologging and not keras autologging.

…lflow#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com>

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy · 2021-09-03T04:59:52Z

mlflow/utils/import_hooks/__init__.py

@@ -247,7 +247,7 @@ def __init__(self):

    @synchronized(_post_import_hooks_lock)
    @synchronized(_import_error_hooks_lock)
-    def find_module(self, fullname, path=None):
+    def find_spec(self, fullname, path, target=None):


@dbczumar Confirmed tensorflow.keras now refers to site-packages/keras/api/_v2/keras/__init__.py

from mlflow.utils.import_hooks import register_post_import_hook register_post_import_hook(lambda x: x, "keras") import tensorflow.keras print(tensorflow.keras)

output:

<module 'tensorflow.keras' from '/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/mlflow-dev-env/lib/python3.7/site-packages/keras/api/_v2/keras/__init__.py'>

Woohoo! Awesome RCA, Haru! Really spectacular!

Signed-off-by: dbczumar <corey.zumar@databricks.com>

tests/tensorflow_autolog/test_tensorflow2_autolog.py

Signed-off-by: dbczumar <corey.zumar@databricks.com>

tests/tensorflow_autolog/test_tensorflow2_autolog.py

Signed-off-by: dbczumar <corey.zumar@databricks.com>

tests/tensorflow_autolog/test_tensorflow2_autolog.py

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy

LGTM!

Signed-off-by: dbczumar <corey.zumar@databricks.com>

WeichenXu123

LGTM.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

WeichenXu123 · 2021-09-03T14:23:17Z

mlflow/tracking/fluent.py

+                    " Keras library import: %s",
+                    str(e),
+                )
+                raise


I add raise here, otherwise it will break the test_universal_autolog_throws_if_specific_autolog_throws_in_test_mode test.

WeichenXu123 · 2021-09-03T14:24:40Z

mlflow/tracking/fluent.py

+        # As a result, Keras autologging must call `mlflow.tensorflow.autolog()` in Keras >= 2.6.0.
+        # Accordingly, we insert this check to ensure that importing tensorflow, which may import
+        # keras, does not enable tensorflow autologging twice.
+        if not TF_AUTOLOG_SETUP_CALLED:


Don't use if autologging_is_disabled(...) here, otherwise it will break test test_autolog_success_message_obeys_disabled

WeichenXu123

I fix broken tests, and also fix 2 bugs. LGTM

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com> * Include keras conditionally Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix root cause Signed-off-by: dbczumar <corey.zumar@databricks.com> * docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tests Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fix 2 Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove keras change Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use is Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove unused modules Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lint fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Try preserve find module Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * fix2 Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Simplify fluent test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tweaks, add a warning Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test excludee Signed-off-by: dbczumar <corey.zumar@databricks.com> * Reverts Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com>

* Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com> * Include keras conditionally Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix root cause Signed-off-by: dbczumar <corey.zumar@databricks.com> * docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tests Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fix 2 Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove keras change Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use is Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove unused modules Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lint fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Try preserve find module Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * fix2 Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Simplify fluent test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tweaks, add a warning Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test excludee Signed-off-by: dbczumar <corey.zumar@databricks.com> * Reverts Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com> Signed-off-by: benwilson <benjamin.wilson@databricks.com>

* WIP build Signed-off-by: benwilson <benjamin.wilson@databricks.com> * Finish unit tests and address prophet build dependencies for serving Signed-off-by: benwilson <benjamin.wilson@databricks.com> * refactor example for prophet and add tracking validation test Signed-off-by: benwilson <benjamin.wilson@databricks.com> * adjusting build env Signed-off-by: benwilson <benjamin.wilson@databricks.com> * linting Signed-off-by: benwilson <benjamin.wilson@databricks.com> * Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. * Enable auto dependency inference in spark flavor (#4759) * use infer_pip_requirements to spark.py Signed-off-by: harupy <hkawamura0130@gmail.com> * fix tests Signed-off-by: harupy <hkawamura0130@gmail.com> * fix tests Signed-off-by: harupy <hkawamura0130@gmail.com> * rename test Signed-off-by: harupy <hkawamura0130@gmail.com> * workaround for databricks Signed-off-by: harupy <hkawamura0130@gmail.com> * workaround for pyspark in databricks Signed-off-by: harupy <hkawamura0130@gmail.com> * rename test Signed-off-by: harupy <hkawamura0130@gmail.com> * address comments Signed-off-by: harupy <hkawamura0130@gmail.com> * better error message Signed-off-by: harupy <hkawamura0130@gmail.com> * fix error message Signed-off-by: harupy <hkawamura0130@gmail.com> * Fix `test_autolog_emits_warning_message_when_model_prediction_fails` (#4768) * Use AttributeError Signed-off-by: harupy <hkawamura0130@gmail.com> * comment Signed-off-by: harupy <hkawamura0130@gmail.com> * fix Signed-off-by: harupy <hkawamura0130@gmail.com> * use sanity_checking (#4767) Signed-off-by: harupy <hkawamura0130@gmail.com> * refactor: Extract the docker image building _build_image_from_context function (#4769) * refactor: Extract the docker image building _build_image_from_context function Signed-off-by: Alexey Volkov <alexey.volkov@ark-kun.com> * blacken Signed-off-by: harupy <hkawamura0130@gmail.com> Co-authored-by: harupy <hkawamura0130@gmail.com> * Fix autologging compatibility with Keras >= 2.6.0 (#4766) * Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com> * Include keras conditionally Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix root cause Signed-off-by: dbczumar <corey.zumar@databricks.com> * docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tests Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fix 2 Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove keras change Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use is Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove unused modules Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lint fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Try preserve find module Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * fix2 Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Simplify fluent test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tweaks, add a warning Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test excludee Signed-off-by: dbczumar <corey.zumar@databricks.com> * Reverts Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com> * Update spark support version to be 3.2 (#4770) * init Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Fix set matrix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> * merge * merge * fix build errors * Changelog message tweak (#4777) * Tweak Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lang Signed-off-by: dbczumar <corey.zumar@databricks.com> * Update CHANGELOG.rst * [ALL TESTS] Update (#4778) * Update MLflow version to 1.20.3 Signed-off-by: Jenkins <jenkins@databricks.com> * Update Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: Jenkins <jenkins@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> * Fix small typo (#4772) Before: ``` 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: precision_score failed. The metric test_gold_precision_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: recall_score failed. The metric test_gold_recall_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: f1_score failed. The metric test_gold_f1_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: accuracy_score failed. The metric test_gold_accuracy_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' ``` Signed-off-by: Louis Guitton <louisguitton93@gmail.com> * Push MLflow model to Sagemaker model registry (#4669) * Added push_sagemaker_model() api to enable push model from MLflow to Sagemaker model registry Signed-off-by: Jinni Gu <jinnigu@uw.edu> * Fixed the python doc for _find_transform_job() Signed-off-by: Jinni Gu <jinnigu@uw.edu> * Replace https protocol with s3:// protocol for the URL where Sagemaker model artifacts are stored in S3. Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * rename push_sagemaker_model function add CLI function for push_model_to_sagemaker Signed-off-by: qtz123 <qiutingzhi1995@gmail.com> * reformat CLI for push_model_to_sagemaker Signed-off-by: qtz123 <qiutingzhi1995@gmail.com> * Used describe_model instead of list_models to check if the model exists. Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * Renamed the _find_model method to _does_model_exist and limit the scope Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * Fixed lint error and changed command to push model Signed-off-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: qtz123 <qiutingzhi1995@gmail.com> * Fix tensorflow dev cross test (#4780) * init Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * dummy tf update Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Adding requirements for prophet examples Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * update conda versions for prophet Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Fixing the prophet tests Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * PR fixes Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Add notes on sub-dependencies Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Updating docs for Prophet flavor Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Resolve dependencies and add flavor tests Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * update conda.yaml in examples for prophet Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com> Co-authored-by: dbczumar <39497902+dbczumar@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: mlflow-automation <61449322+mlflow-automation@users.noreply.github.com> Co-authored-by: Jenkins <jenkins@databricks.com> Co-authored-by: Louis Guitton <louisguitton@users.noreply.github.com> Co-authored-by: Jinni Gu <jinnigu@uw.edu> Co-authored-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: qtz123 <qiutingzhi1995@gmail.com>

* WIP build Signed-off-by: benwilson <benjamin.wilson@databricks.com> * Finish unit tests and address prophet build dependencies for serving Signed-off-by: benwilson <benjamin.wilson@databricks.com> * refactor example for prophet and add tracking validation test Signed-off-by: benwilson <benjamin.wilson@databricks.com> * adjusting build env Signed-off-by: benwilson <benjamin.wilson@databricks.com> * linting Signed-off-by: benwilson <benjamin.wilson@databricks.com> * Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (mlflow#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. * Enable auto dependency inference in spark flavor (mlflow#4759) * use infer_pip_requirements to spark.py Signed-off-by: harupy <hkawamura0130@gmail.com> * fix tests Signed-off-by: harupy <hkawamura0130@gmail.com> * fix tests Signed-off-by: harupy <hkawamura0130@gmail.com> * rename test Signed-off-by: harupy <hkawamura0130@gmail.com> * workaround for databricks Signed-off-by: harupy <hkawamura0130@gmail.com> * workaround for pyspark in databricks Signed-off-by: harupy <hkawamura0130@gmail.com> * rename test Signed-off-by: harupy <hkawamura0130@gmail.com> * address comments Signed-off-by: harupy <hkawamura0130@gmail.com> * better error message Signed-off-by: harupy <hkawamura0130@gmail.com> * fix error message Signed-off-by: harupy <hkawamura0130@gmail.com> * Fix `test_autolog_emits_warning_message_when_model_prediction_fails` (mlflow#4768) * Use AttributeError Signed-off-by: harupy <hkawamura0130@gmail.com> * comment Signed-off-by: harupy <hkawamura0130@gmail.com> * fix Signed-off-by: harupy <hkawamura0130@gmail.com> * use sanity_checking (mlflow#4767) Signed-off-by: harupy <hkawamura0130@gmail.com> * refactor: Extract the docker image building _build_image_from_context function (mlflow#4769) * refactor: Extract the docker image building _build_image_from_context function Signed-off-by: Alexey Volkov <alexey.volkov@ark-kun.com> * blacken Signed-off-by: harupy <hkawamura0130@gmail.com> Co-authored-by: harupy <hkawamura0130@gmail.com> * Fix autologging compatibility with Keras >= 2.6.0 (mlflow#4766) * Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (mlflow#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com> * Include keras conditionally Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix root cause Signed-off-by: dbczumar <corey.zumar@databricks.com> * docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Some test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tests Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test fix 2 Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove keras change Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use is Signed-off-by: dbczumar <corey.zumar@databricks.com> * Remove unused modules Signed-off-by: dbczumar <corey.zumar@databricks.com> * Use fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Docstring Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix fixtures Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lint fixes Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Try preserve find module Signed-off-by: dbczumar <corey.zumar@databricks.com> * fix Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * fix2 Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Simplify fluent test cases Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> * Tweaks, add a warning Signed-off-by: dbczumar <corey.zumar@databricks.com> * Test excludee Signed-off-by: dbczumar <corey.zumar@databricks.com> * Reverts Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com> * Update spark support version to be 3.2 (mlflow#4770) * init Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Fix set matrix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Fix Signed-off-by: dbczumar <corey.zumar@databricks.com> * Format Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> * merge * merge * fix build errors * Changelog message tweak (mlflow#4777) * Tweak Signed-off-by: dbczumar <corey.zumar@databricks.com> * Lang Signed-off-by: dbczumar <corey.zumar@databricks.com> * Update CHANGELOG.rst * [ALL TESTS] Update (mlflow#4778) * Update MLflow version to 1.20.3 Signed-off-by: Jenkins <jenkins@databricks.com> * Update Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: Jenkins <jenkins@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> * Fix small typo (mlflow#4772) Before: ``` 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: precision_score failed. The metric test_gold_precision_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: recall_score failed. The metric test_gold_recall_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: f1_score failed. The metric test_gold_f1_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' 2021/09/03 18:53:45 WARNING mlflow.sklearn.utils: accuracy_score failed. The metric test_gold_accuracy_scorewill not be recorded. Metric error: '<' not supported between instances of 'float' and 'str' ``` Signed-off-by: Louis Guitton <louisguitton93@gmail.com> * Push MLflow model to Sagemaker model registry (mlflow#4669) * Added push_sagemaker_model() api to enable push model from MLflow to Sagemaker model registry Signed-off-by: Jinni Gu <jinnigu@uw.edu> * Fixed the python doc for _find_transform_job() Signed-off-by: Jinni Gu <jinnigu@uw.edu> * Replace https protocol with s3:// protocol for the URL where Sagemaker model artifacts are stored in S3. Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * rename push_sagemaker_model function add CLI function for push_model_to_sagemaker Signed-off-by: qtz123 <qiutingzhi1995@gmail.com> * reformat CLI for push_model_to_sagemaker Signed-off-by: qtz123 <qiutingzhi1995@gmail.com> * Used describe_model instead of list_models to check if the model exists. Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * Renamed the _find_model method to _does_model_exist and limit the scope Signed-off-by: Yiqing Wang <yiqing@wangemail.com> * Fixed lint error and changed command to push model Signed-off-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: qtz123 <qiutingzhi1995@gmail.com> * Fix tensorflow dev cross test (mlflow#4780) * init Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * dummy tf update Signed-off-by: Weichen Xu <weichen.xu@databricks.com> * Adding requirements for prophet examples Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * update conda versions for prophet Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Fixing the prophet tests Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * PR fixes Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Add notes on sub-dependencies Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Updating docs for Prophet flavor Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * Resolve dependencies and add flavor tests Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> * update conda.yaml in examples for prophet Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: jinzhang21 <78067366+jinzhang21@users.noreply.github.com> Co-authored-by: Harutaka Kawamura <hkawamura0130@gmail.com> Co-authored-by: Alexey Volkov <alexey.volkov@ark-kun.com> Co-authored-by: dbczumar <39497902+dbczumar@users.noreply.github.com> Co-authored-by: Weichen Xu <weichen.xu@databricks.com> Co-authored-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: mlflow-automation <61449322+mlflow-automation@users.noreply.github.com> Co-authored-by: Jenkins <jenkins@databricks.com> Co-authored-by: Louis Guitton <louisguitton@users.noreply.github.com> Co-authored-by: Jinni Gu <jinnigu@uw.edu> Co-authored-by: Yiqing Wang <yiqing@wangemail.com> Co-authored-by: qtz123 <qiutingzhi1995@gmail.com> Signed-off-by: Anjali Samad <samad.anjali.14@gmail.com>

dbczumar commented Sep 3, 2021

View reviewed changes

jinzhang21 and others added 4 commits September 2, 2021 21:27

Increase HTTP timeout to 90s. Disabled cloud storage HTTP timeout. (m…

9c02a80

…lflow#4764) * Increase HTTP timeout to 120s. Disabled cloud storage HTTP timeout. Signed-off-by: dbczumar <corey.zumar@databricks.com>

Include keras conditionally

bbcc82b

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Fixes

7ed5725

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Fix

f51cb0b

Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar force-pushed the keras_fix branch from c334b9b to f51cb0b Compare September 3, 2021 04:27

Merge remote-tracking branch 'origin/master' into keras_fix

0983d71

Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar force-pushed the keras_fix branch from e60d5f0 to 0983d71 Compare September 3, 2021 04:28

dbczumar added 2 commits September 2, 2021 21:52

Fix root cause

fd70978

Signed-off-by: dbczumar <corey.zumar@databricks.com>

docstring

17f53bb

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy reviewed Sep 3, 2021

View reviewed changes

harupy approved these changes Sep 3, 2021

View reviewed changes

dbczumar added 4 commits September 2, 2021 22:22

Some test cases

4a50aab

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Some test cases

831ebc2

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Tests

2ce92c6

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Format

edf8760

Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar changed the title ~~Keras fix~~ Fix compatibility with Keras >= 2.6.0 Sep 3, 2021

dbczumar added 2 commits September 2, 2021 23:36

Test fixes

8f5bce4

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Test fix 2

31a5c02

Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar changed the title ~~Fix compatibility with Keras >= 2.6.0~~ Fix autologging compatibility with Keras >= 2.6.0 Sep 3, 2021

github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/bug-fix Mention under Bug Fixes in Changelogs. labels Sep 3, 2021

harupy reviewed Sep 3, 2021

View reviewed changes

tests/tensorflow_autolog/test_tensorflow2_autolog.py Outdated Show resolved Hide resolved

dbczumar added 2 commits September 2, 2021 23:54

Remove keras change

6b646f0

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Use is

d5b45b0

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy reviewed Sep 3, 2021

View reviewed changes

tests/tensorflow_autolog/test_tensorflow2_autolog.py Outdated Show resolved Hide resolved

Remove unused modules

5d06db4

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy reviewed Sep 3, 2021

View reviewed changes

tests/tensorflow_autolog/test_tensorflow2_autolog.py Outdated Show resolved Hide resolved

Use fixtures

4ec1bf7

Signed-off-by: dbczumar <corey.zumar@databricks.com>

harupy approved these changes Sep 3, 2021

View reviewed changes

dbczumar added 5 commits September 3, 2021 00:04

Docstring

b0ce97c

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Fix fixtures

c86883f

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Lint fixes

f195d2f

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Format

b65a2f2

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Try preserve find module

d9cf2ca

Signed-off-by: dbczumar <corey.zumar@databricks.com>

WeichenXu123 approved these changes Sep 3, 2021

View reviewed changes

fix

6f2aac5

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

WeichenXu123 reviewed Sep 3, 2021

View reviewed changes

WeichenXu123 approved these changes Sep 3, 2021

View reviewed changes

WeichenXu123 and others added 6 commits September 4, 2021 01:31

fix2

076f650

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

Simplify fluent test cases

0c88bcf

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Format

29d2e99

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Tweaks, add a warning

a775965

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Test excludee

9e32e6a

Signed-off-by: dbczumar <corey.zumar@databricks.com>

Reverts

6219c3a

Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar merged commit 7cf957b into mlflow:master Sep 3, 2021

WeichenXu123 mentioned this pull request Nov 4, 2021

Fix tensorflow 2.5.2 cross test failure #4995

Merged

27 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix autologging compatibility with Keras >= 2.6.0 #4766

Fix autologging compatibility with Keras >= 2.6.0 #4766

dbczumar commented Sep 3, 2021 •

edited

github-actions bot commented Sep 3, 2021

dbczumar Sep 3, 2021

dbczumar Sep 3, 2021

harupy Sep 3, 2021 •

edited

dbczumar Sep 3, 2021

harupy left a comment

WeichenXu123 left a comment

WeichenXu123 Sep 3, 2021

WeichenXu123 Sep 3, 2021

WeichenXu123 left a comment

		if autologging_is_disabled("tensorflow"):
		setup_autologging(tensorflow_module)

Fix autologging compatibility with Keras >= 2.6.0 #4766

Fix autologging compatibility with Keras >= 2.6.0 #4766

Conversation

dbczumar commented Sep 3, 2021 • edited

What changes are proposed in this pull request?

How is this patch tested?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

github-actions bot commented Sep 3, 2021

dbczumar Sep 3, 2021

Choose a reason for hiding this comment

dbczumar Sep 3, 2021

Choose a reason for hiding this comment

harupy Sep 3, 2021 • edited

Choose a reason for hiding this comment

dbczumar Sep 3, 2021

Choose a reason for hiding this comment

harupy left a comment

Choose a reason for hiding this comment

WeichenXu123 left a comment

Choose a reason for hiding this comment

WeichenXu123 Sep 3, 2021

Choose a reason for hiding this comment

WeichenXu123 Sep 3, 2021

Choose a reason for hiding this comment

WeichenXu123 left a comment

Choose a reason for hiding this comment

dbczumar commented Sep 3, 2021 •

edited

harupy Sep 3, 2021 •

edited