Support dump and load timm and hf_text models on MultiModalPredictor #2682

suzhoum · 2023-01-12T16:33:18Z

Issue #, if available:

Description of changes:

Added APIs to dump timm_image and hf_text models
Added support to load model from path for timm_image

Usage:

To dump models from fine-tuned MultiModalPredictor:

  MultiModalPredictor.dump_timm_image(PATH_TO_MODEL)
  MultiModalPredictor.dump_hf_text(PATH_TO_MODEL)

To load models from saved model path:

hyperparameters = {
     "model.hf_text.checkpoint_name": PATH_TO_SAVED_HF_TEXT_MODEL,
     "model.timm_image.checkpoint_name": PATH_TO_SAVED_TIMM_IMAGE_MODEL,
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions · 2023-01-12T20:49:42Z

Job PR-2682-1adf0a6 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2682/1adf0a6/index.html

multimodal/src/autogluon/multimodal/models/timm_image.py

multimodal/src/autogluon/multimodal/utils/model.py

zhiqiangdon · 2023-01-13T01:25:24Z

multimodal/src/autogluon/multimodal/predictor.py

+        if isinstance(self._model, MultimodalFusionMLP) and isinstance(
+            self._model.model, torch.nn.modules.container.ModuleList
+        ):
+            for per_model in self._model.model:
+                if isinstance(per_model, TimmAutoModelForImagePrediction):
+                    model = per_model
+                    break


Saving a timm image backbone from a fusion model seems not useful since it can't work individually. We can consider only deal with single timm model for now.

We considered the scenario for fusion model because there might be a use case that a user wants to save specifically timm_image or hf_text from the fusion model trained to use on other downstream tasks which only use a single model. Do you feel that would be useful?

For example, people trained on the mixture of image, text, tabular data, and like to extract the image part from the fusion model. The API “predictor.dump_timm_image” should still work in this case.

This logic has the limitation that, If multiple timm image models are available, it would only dump the first one.

I've updated the logic to support saving multiple models under timm_image and hf_text.

zhiqiangdon · 2023-01-13T01:35:11Z

multimodal/src/autogluon/multimodal/predictor.py

+        model = None
+        if isinstance(self._model, MultimodalFusionMLP) and isinstance(
+            self._model.model, torch.nn.modules.container.ModuleList
+        ):
+            for per_model in self._model.model:
+                if isinstance(per_model, HFAutoModelForTextPrediction):
+                    model = per_model
+                    break


Ditto. We can consider only the single huggingface text model for now.

zhiqiangdon · 2023-01-13T01:36:23Z

multimodal/src/autogluon/multimodal/predictor.py

+            os.makedirs(path)
+            model.model.save_pretrained(path)
+            logger.info(f"Model saved to {path}.")
+            if TEXT in self._data_processors.keys():


For hf_text model, we can assert self._data_processors have only one text processor.

Do you mean there is no need to check the prefix of the data processor to get the tokenizer?

Since we consider dumping a hf model from a fusion model, this logic is OK.

Do we need raise error if there is no hf model available?

zhiqiangdon · 2023-01-13T01:45:45Z

multimodal/src/autogluon/multimodal/predictor.py

@@ -2812,6 +2817,79 @@ def load(

        return predictor

+    def dump_timm_image(


There would be too many APIs if each model has a dump function. How about using only one API dump_model()? Inside the function, we can check the model type is timm image or hf text.

zhiqiangdon · 2023-01-13T01:46:18Z

multimodal/src/autogluon/multimodal/utils/model.py

+    return filtered_cfg
+
+
+def save_timm_config(


docstrings are missing.

sxjscience · 2023-01-13T16:25:43Z

multimodal/src/autogluon/multimodal/predictor.py

+
+        model = self._model if model is None else model
+        if isinstance(model, HFAutoModelForTextPrediction) and model.model is not None:
+            os.makedirs(path)


Try to use “os.makedirs(path, exist_ok=True)”.

I originally had this flag set, but after discussing with Zhiqiang offline, he suggested that we might not want to save to an existing (or non-empty) directory to avoid accidentally overwriting the model.

github-actions · 2023-01-17T21:44:10Z

Job PR-2682-303d91d is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2682/303d91d/index.html

github-actions · 2023-01-17T21:51:19Z

Job PR-2682-b289c4a is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2682/b289c4a/index.html

zhiqiangdon · 2023-01-17T22:35:03Z

multimodal/src/autogluon/multimodal/utils/model.py

@@ -523,3 +524,47 @@ def modify_duplicate_model_names(

 def list_timm_models(pretrained=True):
    return timm.list_models(pretrained=pretrained)
+
+
+def _filter_timm_pretrained_cfg(cfg, remove_source=False, remove_null=True):


Since _filter_timm_pretrained_cfg and save_timm_config are about config, is it better to put them into utils/config.py?

zhiqiangdon · 2023-01-17T22:35:34Z

multimodal/tests/unittests/predictor/test_predictor.py

+    timm_image_dir = f"{model_dump_path}/timm_image"
+    assert os.path.exists(hf_text_dir) and (len(os.listdir(hf_text_dir)) > 2) == True
+    assert os.path.exists(timm_image_dir) and (len(os.listdir(timm_image_dir)) == 2) == True
+    print("done")


Remove print?

zhiqiangdon · 2023-01-17T22:47:55Z

multimodal/src/autogluon/multimodal/predictor.py

+        path : str
+            Path to directory where models and configs should be saved.
+        """
+        models = {}


models are list and dict in dump_timm_image and dump_hf_text , respectively. Can we use list for both of them?

zhiqiangdon · 2023-01-17T22:50:24Z

multimodal/src/autogluon/multimodal/predictor.py

+            Path to directory where models and configs should be saved.
+        """
+        models = []
+        if isinstance(self._model, MultimodalFusionMLP) and isinstance(


A fusion model may also be MultimodalFusionTransformer.

github-actions · 2023-01-18T08:13:24Z

Job PR-2682-f7801b7 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2682/f7801b7/index.html

zhiqiangdon

LGTM! Awesome feature! Consider unifying the dumping functions into one API in follow-up PRs.

sxjscience · 2023-01-18T21:25:35Z

multimodal/src/autogluon/multimodal/predictor.py

+            Path to directory where models and configs should be saved.
+        """
+        models = []
+        if isinstance(self._model, (MultimodalFusionMLP, MultimodalFusionTransformer)) and isinstance(


@zhiqiangdon We can later add a BaseMultimodalFusionModel class, and ensure that MultimodalFusionMLP and MultimodalFusionTransformer inherit from this class.

@suzhoum May be we can add a TODO item here

Sure. We can add a base class.

sxjscience · 2023-01-18T21:28:27Z

multimodal/tests/unittests/predictor/test_predictor.py

@@ -679,3 +681,104 @@ def test_image_bytearray():
    npt.assert_array_equal(
        [prediction_prob_1, prediction_prob_2, prediction_prob_3, prediction_prob_4], [prediction_prob_1] * 4
    )
+
+
+def test_dump_timm_image():


Can we separate these tests to another file? The reason is that test_predictor.py is growing to be too huge and too slow.

We can consider to add it under the following name

unittests/predictor/test_predictor_dump_third_party.py

Good idea. Done!

sxjscience

Two minor comments!! Overall LGTM!!!! Thanks!!!

github-actions · 2023-01-18T23:41:48Z

Job PR-2682-771fb04 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2682/771fb04/index.html

suzhoum force-pushed the continual_learning branch from 408839d to 4ea1a19 Compare January 12, 2023 16:40

suzhoum requested review from zhiqiangdon and sxjscience January 12, 2023 17:36

suzhoum force-pushed the continual_learning branch from 4ea1a19 to 1adf0a6 Compare January 12, 2023 19:04

zhiqiangdon reviewed Jan 13, 2023

View reviewed changes

multimodal/src/autogluon/multimodal/models/timm_image.py Show resolved Hide resolved

zhiqiangdon reviewed Jan 13, 2023

View reviewed changes

multimodal/src/autogluon/multimodal/utils/model.py Outdated Show resolved Hide resolved

zhiqiangdon reviewed Jan 13, 2023

View reviewed changes

suzhoum force-pushed the continual_learning branch from 1adf0a6 to ad79e84 Compare January 13, 2023 06:54

sxjscience reviewed Jan 13, 2023

View reviewed changes

suzhoum force-pushed the continual_learning branch 2 times, most recently from ce8ebb7 to 303d91d Compare January 17, 2023 19:57

suzhoum added 4 commits January 17, 2023 20:07

add dump_timm_image and dump_hf_text to MultiModalPredictor

fb480b8

load timm image model from path when supplied in hyperparameter

5052249

add tests

3f163dd

support multiple models under fusion_mlp

b289c4a

suzhoum force-pushed the continual_learning branch from 303d91d to b289c4a Compare January 17, 2023 20:08

zhiqiangdon reviewed Jan 17, 2023

View reviewed changes

address CR comments

f7801b7

suzhoum requested a review from zhiqiangdon January 18, 2023 17:59

zhiqiangdon approved these changes Jan 18, 2023

View reviewed changes

sxjscience reviewed Jan 18, 2023

View reviewed changes

sxjscience approved these changes Jan 18, 2023

View reviewed changes

refactor

771fb04

suzhoum merged commit ab0f8cf into autogluon:master Jan 18, 2023

suzhoum added this to the 0.7 Release milestone Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support dump and load timm and hf_text models on MultiModalPredictor #2682

Support dump and load timm and hf_text models on MultiModalPredictor #2682

suzhoum commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

zhiqiangdon Jan 13, 2023

suzhoum Jan 13, 2023

sxjscience Jan 13, 2023

zhiqiangdon Jan 13, 2023

suzhoum Jan 17, 2023

zhiqiangdon Jan 13, 2023

zhiqiangdon Jan 13, 2023 •

edited

suzhoum Jan 13, 2023

zhiqiangdon Jan 13, 2023

zhiqiangdon Jan 13, 2023

zhiqiangdon Jan 13, 2023

zhiqiangdon Jan 13, 2023 •

edited

sxjscience Jan 13, 2023

suzhoum Jan 13, 2023 •

edited

github-actions bot commented Jan 17, 2023

github-actions bot commented Jan 17, 2023

zhiqiangdon Jan 17, 2023

zhiqiangdon Jan 17, 2023

zhiqiangdon Jan 17, 2023

zhiqiangdon Jan 17, 2023

github-actions bot commented Jan 18, 2023

zhiqiangdon left a comment

sxjscience Jan 18, 2023

sxjscience Jan 18, 2023

zhiqiangdon Jan 18, 2023

sxjscience Jan 18, 2023

suzhoum Jan 18, 2023

sxjscience left a comment

github-actions bot commented Jan 18, 2023

		@@ -2812,6 +2817,79 @@ def load(

		return predictor

		def dump_timm_image(

Support dump and load timm and hf_text models on MultiModalPredictor #2682

Support dump and load timm and hf_text models on MultiModalPredictor #2682

Conversation

suzhoum commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiqiangdon Jan 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiqiangdon Jan 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suzhoum Jan 13, 2023 • edited

Choose a reason for hiding this comment

github-actions bot commented Jan 17, 2023

github-actions bot commented Jan 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 18, 2023

zhiqiangdon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sxjscience left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 18, 2023

zhiqiangdon Jan 13, 2023 •

edited

zhiqiangdon Jan 13, 2023 •

edited

suzhoum Jan 13, 2023 •

edited