[Data] Add `Dataset.write_images` #38228

bveeramani · 2023-08-08T19:17:09Z

Why are these changes needed?

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

amogkam · 2023-08-08T19:39:32Z

Let's tag this as an alpha API and make the necessary changes to the documentation (API reference, working with images)?

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

python/ray/data/dataset.py

python/ray/data/datasource/file_based_datasource.py

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

…o write-images Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

python/ray/data/datasource/file_based_datasource.py

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

bveeramani · 2023-08-09T02:16:48Z

Failing code format is unrelated:



dashboard/modules/reporter/reporter_head.py:42:89: E501 line too long (91 > 88 characters)
--
  | dashboard/modules/reporter/reporter_head.py:188:89: E501 line too long (95 > 88 characters)
  | dashboard/modules/reporter/reporter_head.py:191:30: F541 f-string is missing placeholders

c21 · 2023-08-09T04:10:46Z

python/ray/data/dataset.py

+    def write_images(
+        self,
+        path: str,
+        column: str,


why we require users to set column by default? Should this be optional? By default we should assume the dataset should only have one column, and users want to write it as an image, right?

c21 · 2023-08-09T04:15:11Z

python/ray/data/datasource/image_datasource.py

+
+        image = Image.fromarray(row[column])
+        buffer = io.BytesIO()
+        image.save(buffer, format=file_format)


What's the error message looks like if user provides an illegal file format?

c21 · 2023-08-09T04:19:14Z

python/ray/data/tests/test_image.py

+    @pytest.mark.parametrize("file_format", [None, "png"])
+    def test_write_images(ray_start_regular_shared, file_format, tmp_path):
+        ds = ray.data.read_images("example://image-datasets/simple")
+        ds.write_images(


Can we also test followed cases:

the dataset has multiple columns

the provided file format is illegal

write_images(), then read_images() back.

c21 · 2023-08-09T04:21:20Z

python/ray/data/dataset.py

+            path: The path to the destination root directory, where
+                the images are written to.
+            column: The column containing the data you want to write to images.
+            file_format: The image file format to write with. For available options,


it's good we provide a link here, let's do one more step further, just to give users a list of common file formats, so they don't need to read the other documentation:

Examples of popular file formats are "png", "jpg", "jpeg", "tif", "tiff", "bmp", "gif". By default write in "png" format.

#37043, #37874, and #38228 broke the code-format lint. This PR fixes it.

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API. Signed-off-by: NripeshN <nn2012@hw.ac.uk>

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it. Signed-off-by: NripeshN <nn2012@hw.ac.uk>

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API. Signed-off-by: harborn <gangsheng.wu@intel.com>

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it. Signed-off-by: harborn <gangsheng.wu@intel.com>

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API.

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it.

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API. Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it. Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API. Signed-off-by: Victor <vctr.y.m@example.com>

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it. Signed-off-by: Victor <vctr.y.m@example.com>

Initial commit

d375743

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

bveeramani requested review from ericl, scv119, c21, amogkam, scottjlee and raulchen as code owners August 8, 2023 19:17

Add write_images

7e37f7b

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

bveeramani changed the title ~~[Data] Add support for writing images~~ [Data] Add Dataset.write_images` Aug 8, 2023

bveeramani changed the title ~~[Data] Add Dataset.write_images`~~ [Data] Add Dataset.write_images Aug 8, 2023

bveeramani assigned amogkam Aug 8, 2023

bveeramani added 2 commits August 8, 2023 15:04

Address review comments

75f7a0c

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Merge branch 'master' into write-images

6b6b3ca

amogkam reviewed Aug 8, 2023

View reviewed changes

python/ray/data/dataset.py Outdated Show resolved Hide resolved

python/ray/data/dataset.py Outdated Show resolved Hide resolved

python/ray/data/datasource/file_based_datasource.py Show resolved Hide resolved

python/ray/data/datasource/file_based_datasource.py Outdated Show resolved Hide resolved

bveeramani added 6 commits August 8, 2023 16:56

Address review comments

450d16c

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Merge branch 'write-images' of https://github.com/ray-project/ray int…

caad5f5

…o write-images Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Update file_based_datasource.py

41c80b8

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Update image_datasource.py

3e7829c

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Update file_based_datasource.py

32abd49

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

Update file_based_datasource.py

cdcaf09

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

amogkam approved these changes Aug 9, 2023

View reviewed changes

python/ray/data/datasource/file_based_datasource.py Outdated Show resolved Hide resolved

python/ray/data/datasource/file_based_datasource.py Show resolved Hide resolved

Update file_based_datasource.py

a98fb7d

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>

bveeramani added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Aug 9, 2023

bveeramani merged commit 273a2a1 into master Aug 9, 2023
59 of 63 checks passed

bveeramani deleted the write-images branch August 9, 2023 02:17

bveeramani mentioned this pull request Aug 9, 2023

[CI] Fix failing code-format lint #38245

Merged

8 tasks

c21 reviewed Aug 9, 2023

View reviewed changes

bveeramani added a commit that referenced this pull request Aug 9, 2023

[CI] Fix failing code-format lint (#38245)

9c193f4

#37043, #37874, and #38228 broke the code-format lint. This PR fixes it.

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[Data] Add Dataset.write_images (ray-project#38228)

31556e0

Users want to write images back to S3 (for example, segmentation masks from batch inference). So, this PR adds a Dataset.write_images API.

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[CI] Fix failing code-format lint (ray-project#38245)

6c40748

ray-project#37043, ray-project#37874, and ray-project#38228 broke the code-format lint. This PR fixes it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Add `Dataset.write_images` #38228

[Data] Add `Dataset.write_images` #38228

bveeramani commented Aug 8, 2023

amogkam commented Aug 8, 2023 •

edited

bveeramani commented Aug 9, 2023

c21 Aug 9, 2023

c21 Aug 9, 2023

c21 Aug 9, 2023

c21 Aug 9, 2023

[Data] Add Dataset.write_images #38228

[Data] Add Dataset.write_images #38228

Conversation

bveeramani commented Aug 8, 2023

Why are these changes needed?

Related issue number

Checks

amogkam commented Aug 8, 2023 • edited

bveeramani commented Aug 9, 2023

c21 Aug 9, 2023

Choose a reason for hiding this comment

c21 Aug 9, 2023

Choose a reason for hiding this comment

c21 Aug 9, 2023

Choose a reason for hiding this comment

c21 Aug 9, 2023

Choose a reason for hiding this comment

[Data] Add `Dataset.write_images` #38228

[Data] Add `Dataset.write_images` #38228

amogkam commented Aug 8, 2023 •

edited