Train and inference with different image data types #2535

suzhoum · 2022-12-08T00:48:22Z

Issue #, if available:

Description of changes:
This PR combines current different image modalities (image_path and image_bytearray) into single image modality. This supports image training and inference with different feature types.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tonyhoo · 2022-12-08T16:11:23Z

multimodal/src/autogluon/multimodal/data/infer_types.py

@@ -131,6 +131,33 @@ def is_rois_column(data: pd.Series) -> bool:
        return is_rois_input(data[idx])


+def is_image_path(feature: Any):
+    is_path = True
+    image_paths = str(feature).split(";")


wondering why we need to split by ";"? Is there any upstream logics introducing it or some assumptions made?

Yes there is an existing upstream logic for image_path that we use ";" to separate a list of images.

tonyhoo · 2022-12-08T18:00:40Z

multimodal/src/autogluon/multimodal/data/preprocess_dataframe.py

+    def _is_image_path(self, feature: Any):
+        is_path = True
+        image_paths = str(feature).split(";")
+        for img_path in image_paths:
+            try:
+                with PIL.Image.open(img_path) as img:
+                    pass
+                break
+            except:
+                is_path = False
+        return is_path
+
+    def _is_image_bytearray(self, feature: Any):
+        is_bytearray = True
+        if not isinstance(feature, list):
+            feature = [feature]
+        for img_bytearray in feature:
+            try:
+                with PIL.Image.open(BytesIO(img_bytearray)) as img:
+                    pass
+                break
+            except:
+                is_bytearray = False
+        return is_bytearray


These 2 functions seems to be duplicate of what defined in infer_types.py. Better to consolidate and reuse

Yes the intention was to add these utility function inside./multimodal/data/utils.py or ./multimodal/utils/data.py, but they both depend on MultiModalFeaturePreprocessor which creates circular dependency. Unless we want to create a new utility function.

can we let infer_types.py use the function defined here?

It would require us to initialize a MultiModalFeaturePreprocessor with variables that are not readily available, and potentially introduce unstable dependencies since there are files that import both infer_types and MultiModalFeaturePreprocessor, e.g.

autogluon/multimodal/src/autogluon/multimodal/predictor.py

Lines 86 to 92 in 0a62d48

from .data.infer_types import (

infer_column_types,

infer_label_column_type_by_problem_type,

infer_problem_type_output_shape,

infer_rois_column_type,

)

from .data.preprocess_dataframe import MultiModalFeaturePreprocessor

…t column types

gradientsky · 2022-12-08T22:29:46Z

multimodal/src/autogluon/multimodal/data/infer_types.py

-    data = data.tolist()
+    image_type = get_image_feature_type(data.iloc[0])
+    if image_type == IMAGE_PATH:
+        data = data.apply(lambda ele: str(ele).split(";")).tolist()


Please use vectorized form: data.str.split(';').tolist()

We need to distinguish image_path and image_bytearray in the column types. These image sub-types would be used in df preprocessor and image processor. The current modification breaks the original logic.

The problem is that there is a difference of usage between training and inference time. We can train using images on disk and then use bytearrays during the inference. Saving files to a file system during the inference time creates additional security overhead.

I agree that we need to support using image_path in training and image_bytearray in inference. To do so, we can infer the image column types during inference and see whether the subtype changes. If we detect subtype changes, we can modify the _column_types in df preprocessor to reflect them. There is no need to change the internal logic among infer_types, df preprocessor, and processor.

Thanks for the suggestion. I'm working on a quick POC for this idea, and hopefully we can still catch the release.

zhiqiangdon

We need to distinguish image_path and image_bytearray in the column types. These image sub-types would be used in df preprocessor and image processor. The current modification breaks the original logic.

gradientsky · 2022-12-08T23:23:19Z

We need to distinguish image_path and image_bytearray in the column types. These image sub-types would be used in df preprocessor and image processor. The current modification breaks the original logic.

The problem is that there is a difference of usage between training and inference time. We can train using images on disk and then use bytearrays during the inference. Saving files to a file system during the inference time creates additional security overhead.

github-actions · 2022-12-08T23:31:28Z

Job PR-2535-b8a52d8 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/b8a52d8/index.html

zhiqiangdon · 2022-12-08T23:44:07Z

We need to distinguish image_path and image_bytearray in the column types. These image sub-types would be used in df preprocessor and image processor. The current modification breaks the original logic.

The problem is that there is a difference of usage between training and inference time. We can train using images on disk and then use bytearrays during the inference. Saving files to a file system during the inference time creates additional security overhead.

I agree that we need to support using image_path in training and image_bytearray in inference. To do so, we can infer the image column types during inference and see whether the subtype changes. If we detect subtype changes, we can modify the _column_types in df preprocessor to reflect them. There is no need to change the internal logic among infer_types, df preprocessor, and processor.

github-actions · 2022-12-09T02:04:59Z

Job PR-2535-b16626f is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/b16626f/index.html

github-actions · 2022-12-09T04:13:52Z

Job PR-2535-b240ce3 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/b240ce3/index.html

multimodal/src/autogluon/multimodal/predictor.py

github-actions · 2022-12-09T04:40:19Z

Job PR-2535-441ce57 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/441ce57/index.html

zhiqiangdon

LGTM! Consider using the updated column_types to explicitly replace the df_preprocessor._column_types to make logic more clear.

tonyhoo · 2022-12-09T07:43:26Z

LGTM! Consider using the updated column_types to explicitly replace the df_preprocessor._column_types to make logic more clear.

Wondering what the current behavior would be if the df_preprocessor._column_types not in sync

github-actions · 2022-12-09T08:13:43Z

Job PR-2535-8cb0dda is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/8cb0dda/index.html

github-actions · 2022-12-09T17:48:19Z

Job PR-2535-725f115 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2535/725f115/index.html

suzhoum force-pushed the single_image_modality branch 3 times, most recently from 2716445 to a182803 Compare December 8, 2022 16:19

tonyhoo reviewed Dec 8, 2022

View reviewed changes

suzhoum force-pushed the single_image_modality branch 3 times, most recently from 809eae3 to acf4464 Compare December 8, 2022 21:10

suzhoum marked this pull request as ready for review December 8, 2022 21:20

suzhoum added 3 commits December 8, 2022 21:22

combine image_path and image_bytearray as image modality

8719640

check image type during image processing

bc293bd

add test coverage for training and inference image data with differen…

1ae77c1

…t column types

suzhoum requested a review from zhiqiangdon December 8, 2022 21:23

refactor

b8a52d8

suzhoum force-pushed the single_image_modality branch from acf4464 to b8a52d8 Compare December 8, 2022 22:15

gradientsky reviewed Dec 8, 2022

View reviewed changes

zhiqiangdon suggested changes Dec 8, 2022

View reviewed changes

suzhoum force-pushed the single_image_modality branch from 3af4ed4 to b16626f Compare December 9, 2022 00:44

suzhoum requested a review from zhiqiangdon December 9, 2022 02:27

suzhoum force-pushed the single_image_modality branch 2 times, most recently from 441ce57 to b240ce3 Compare December 9, 2022 02:55

suzhoum changed the title ~~Single image modality~~ Train and inference with different image data types Dec 9, 2022

zhiqiangdon reviewed Dec 9, 2022

View reviewed changes

multimodal/src/autogluon/multimodal/predictor.py Outdated Show resolved Hide resolved

suzhoum force-pushed the single_image_modality branch from b240ce3 to 8cb0dda Compare December 9, 2022 06:55

zhiqiangdon approved these changes Dec 9, 2022

View reviewed changes

suzhoum force-pushed the single_image_modality branch 2 times, most recently from 437530d to a67ff50 Compare December 9, 2022 07:55

suzhoum force-pushed the single_image_modality branch from a67ff50 to 80ce2d9 Compare December 9, 2022 16:26

suzhoum added 2 commits December 9, 2022 16:27

update image subtype if inference and training don't match

b4a6465

revert

725f115

suzhoum force-pushed the single_image_modality branch from 80ce2d9 to 725f115 Compare December 9, 2022 16:28

tonyhoo approved these changes Dec 9, 2022

View reviewed changes

suzhoum merged commit 96917d1 into autogluon:master Dec 9, 2022

suzhoum added this to the 0.7 Release milestone Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Train and inference with different image data types #2535

Train and inference with different image data types #2535

suzhoum commented Dec 8, 2022

tonyhoo Dec 8, 2022

suzhoum Dec 8, 2022

tonyhoo Dec 8, 2022

suzhoum Dec 8, 2022 •

edited

tonyhoo Dec 8, 2022

suzhoum Dec 8, 2022

gradientsky Dec 8, 2022

suzhoum Dec 9, 2022

zhiqiangdon left a comment

gradientsky commented Dec 8, 2022 •

edited

github-actions bot commented Dec 8, 2022

zhiqiangdon commented Dec 8, 2022

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

zhiqiangdon left a comment

tonyhoo commented Dec 9, 2022 •

edited

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

	from .data.infer_types import (
	infer_column_types,
	infer_label_column_type_by_problem_type,
	infer_problem_type_output_shape,
	infer_rois_column_type,
	)
	from .data.preprocess_dataframe import MultiModalFeaturePreprocessor

Train and inference with different image data types #2535

Train and inference with different image data types #2535

Conversation

suzhoum commented Dec 8, 2022

tonyhoo Dec 8, 2022

Choose a reason for hiding this comment

suzhoum Dec 8, 2022

Choose a reason for hiding this comment

tonyhoo Dec 8, 2022

Choose a reason for hiding this comment

suzhoum Dec 8, 2022 • edited

Choose a reason for hiding this comment

tonyhoo Dec 8, 2022

Choose a reason for hiding this comment

suzhoum Dec 8, 2022

Choose a reason for hiding this comment

gradientsky Dec 8, 2022

Choose a reason for hiding this comment

suzhoum Dec 9, 2022

Choose a reason for hiding this comment

zhiqiangdon left a comment

Choose a reason for hiding this comment

gradientsky commented Dec 8, 2022 • edited

github-actions bot commented Dec 8, 2022

zhiqiangdon commented Dec 8, 2022

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

zhiqiangdon left a comment

Choose a reason for hiding this comment

tonyhoo commented Dec 9, 2022 • edited

github-actions bot commented Dec 9, 2022

github-actions bot commented Dec 9, 2022

suzhoum Dec 8, 2022 •

edited

gradientsky commented Dec 8, 2022 •

edited

tonyhoo commented Dec 9, 2022 •

edited