Support image bytearray in AutoMM #2490

suzhoum · 2022-11-28T19:58:56Z

Issue #, if available:

Description of changes:
This PR adds support for image bytearrays in AutoMM.
The DataFrame column can be either bytearray or list[bytearray].

Benchmarking on the original image_path workflow with dataset on SageMaker endpoint

test data:
100 x bytearray-encoded images

data preprocess:
0.5s to decode and save to disk to generate image_path to pass in MultiModalPredictor.fit()

predict_proba:
0.1s

With the support of bytearray, we can save the time spent on data preprocessing, which majority of the time is on saving the image to disk.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

multimodal/tests/unittests/predictor/test_predictor.py

cheungdaven

LGTM!

bryanyzhu

LGTM with a minor suggestion for unit test.

multimodal/tests/unittests/predictor/test_predictor.py

github-actions · 2022-11-29T08:27:03Z

Job PR-2490-909b7c3 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2490/909b7c3/index.html

multimodal/src/autogluon/multimodal/data/infer_types.py

github-actions · 2022-11-29T19:48:28Z

Job PR-2490-0c15ed6 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2490/0c15ed6/index.html

tonyhoo

Thanks for the change. Some follow up actions in general:

We should identify any doc/tutorial to be updated with this feature as it allows a new input type and we should make it visible via doc to the end user
Can we build docker images for both GPU and CPU to see the inference impact on Sagemaker endpoint? If already done, ignore this point

multimodal/src/autogluon/multimodal/data/infer_types.py

tonyhoo · 2022-11-29T23:40:28Z

multimodal/src/autogluon/multimodal/data/infer_types.py

@@ -156,6 +162,7 @@ def is_numerical_column(
 def is_imagepath_column(
    data: pd.Series,
    col_name: str,
+    sample_n: Optional[int] = 500,


let's add doc string for this field below as well

multimodal/src/autogluon/multimodal/data/infer_types.py

multimodal/src/autogluon/multimodal/data/process_image.py

sxjscience

LGTM

suzhoum · 2022-11-30T03:14:55Z

Thanks for the change. Some follow up actions in general:

We should identify any doc/tutorial to be updated with this feature as it allows a new input type and we should make it visible via doc to the end user

Can we build docker images for both GPU and CPU to see the inference impact on Sagemaker endpoint? If already done, ignore this point

Thanks for the suggestions! We can mention this feature in our tutorials in a follow up PR. The benchmark was done on a GPU instance and I can build a CPU endpoint as well.

github-actions · 2022-11-30T05:42:44Z

Job PR-2490-e973f14 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2490/e973f14/index.html

zhiqiangdon · 2022-11-30T07:17:45Z

multimodal/src/autogluon/multimodal/data/infer_types.py

+
+        for img_bytearray in image_bytearrays:
+            try:
+                with PIL.Image.open(BytesIO(img_bytearray)) as img:


What if the image bytes are encoded by base64?

Can we enforce that the type passed into the predictor to be the original bytearray? In the cloud serve scripts, I see that currently we would: 1. decode the encoded bytearray in a predefined way, and 2. save to disk to get the path to pass into the predictor. With the new feature, we can save step 2 by assuming no encoding on bytearray. If we are to expect encoded bytearrays in the dataframe, we would probably need to accept a callback decoding function provided by the user. What do you think?

base64 is used in encoding images in some cases. I think we'd better support users to provide base64 bytes in dataframe.

base64 is fundamentally a different format than bytearray, since it is a text encoding rather than a binary encoding. We could consider adding a IMAGE_BASE64 column type, or we could try to make a more generic IMAGE_BINARY type that also infers the encoding. But I think this is another feature request separate from the current issue.

Since base64 requires additional decoding, we may define another type IMAGE_BASE64. We can leave this to another PR.

zhiqiangdon · 2022-11-30T07:38:23Z

multimodal/src/autogluon/multimodal/utils/misc.py

+    return [_read_byte(os.path.abspath(os.path.join(base_folder, path))) for path in path_l]
+
+
+def shopee_dataset(


Besides unit tests, where do we use this dataset? It doesn't fit the context of this file.

It is used in a few tutorials at the moment. Do you think creating a datasets.py under the ./utils will help?

We usually put the data preparation logic inside the tutorials, e.g., https://auto.gluon.ai/stable/tutorials/multimodal/multimodal_prediction/beginner_multimodal.html. Users can better understand what the data preparation in this way.

multimodal/src/autogluon/multimodal/data/process_image.py

multimodal/src/autogluon/multimodal/data/infer_types.py

FANGAreNotGnu

LGTM

github-actions · 2022-12-01T18:04:04Z

Job PR-2490-5cce4b8 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2490/5cce4b8/index.html

suzhoum force-pushed the add_bytearray branch from 3ff219a to fa98433 Compare November 28, 2022 20:41

suzhoum changed the title ~~Add bytearray~~ Support image bytearray in AutoMM Nov 28, 2022

suzhoum marked this pull request as ready for review November 28, 2022 21:15

suzhoum requested review from zhiqiangdon, FANGAreNotGnu, cheungdaven, bryanyzhu, sxjscience and tonyhoo November 28, 2022 21:16

bryanyzhu reviewed Nov 28, 2022

View reviewed changes

multimodal/tests/unittests/predictor/test_predictor.py Outdated Show resolved Hide resolved

cheungdaven approved these changes Nov 28, 2022

View reviewed changes

bryanyzhu approved these changes Nov 29, 2022

View reviewed changes

multimodal/tests/unittests/predictor/test_predictor.py Show resolved Hide resolved

suzhoum force-pushed the add_bytearray branch from 326c69f to 909b7c3 Compare November 29, 2022 06:09

sxjscience reviewed Nov 29, 2022

View reviewed changes

multimodal/src/autogluon/multimodal/data/infer_types.py Outdated Show resolved Hide resolved

tonyhoo reviewed Nov 30, 2022

View reviewed changes

sxjscience approved these changes Nov 30, 2022

View reviewed changes

zhiqiangdon reviewed Nov 30, 2022

View reviewed changes

multimodal/src/autogluon/multimodal/data/process_image.py Show resolved Hide resolved

gidler reviewed Nov 30, 2022

View reviewed changes

multimodal/src/autogluon/multimodal/data/infer_types.py Outdated Show resolved Hide resolved

FANGAreNotGnu approved these changes Nov 30, 2022

View reviewed changes

suzhoum force-pushed the add_bytearray branch from c61a3ec to cc982b4 Compare November 30, 2022 23:38

suzhoum added 4 commits November 30, 2022 23:38

check if a column contains image bytearray

83cb350

process image based on modalities

960a1ae

add bytearray as a modality

b9452b7

add bytearray in datasets

a901b8b

suzhoum added 3 commits November 30, 2022 23:38

add test coverage for bytearray

c668021

make sample_num configurable

5ecba28

update docstring

ab6a103

suzhoum force-pushed the add_bytearray branch from cc982b4 to c7b339c Compare November 30, 2022 23:39

tonyhoo approved these changes Dec 1, 2022

View reviewed changes

suzhoum added 2 commits December 1, 2022 15:25

make image_mode configurable

6495d3f

add todo

5cce4b8

suzhoum force-pushed the add_bytearray branch from b082f97 to 5cce4b8 Compare December 1, 2022 15:26

suzhoum merged commit c1fd4db into autogluon:master Dec 1, 2022

suzhoum added this to the 0.7 Release milestone Feb 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support image bytearray in AutoMM #2490

Support image bytearray in AutoMM #2490

suzhoum commented Nov 28, 2022 •

edited

cheungdaven left a comment

bryanyzhu left a comment

github-actions bot commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

tonyhoo left a comment

tonyhoo Nov 29, 2022

sxjscience left a comment

suzhoum commented Nov 30, 2022

github-actions bot commented Nov 30, 2022

zhiqiangdon Nov 30, 2022

suzhoum Nov 30, 2022

zhiqiangdon Nov 30, 2022

gidler Nov 30, 2022

zhiqiangdon Dec 1, 2022

zhiqiangdon Nov 30, 2022

suzhoum Nov 30, 2022

zhiqiangdon Dec 1, 2022 •

edited

FANGAreNotGnu left a comment

github-actions bot commented Dec 1, 2022

		return [_read_byte(os.path.abspath(os.path.join(base_folder, path))) for path in path_l]


		def shopee_dataset(

Support image bytearray in AutoMM #2490

Support image bytearray in AutoMM #2490

Conversation

suzhoum commented Nov 28, 2022 • edited

cheungdaven left a comment

Choose a reason for hiding this comment

bryanyzhu left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 29, 2022

github-actions bot commented Nov 29, 2022

tonyhoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sxjscience left a comment

Choose a reason for hiding this comment

suzhoum commented Nov 30, 2022

github-actions bot commented Nov 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiqiangdon Dec 1, 2022 • edited

Choose a reason for hiding this comment

FANGAreNotGnu left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 1, 2022

suzhoum commented Nov 28, 2022 •

edited

zhiqiangdon Dec 1, 2022 •

edited