refactor video processor (part # 7776) #7861

yiyixuxu · 2024-05-05T10:15:56Z

part of #7776

I refactored the preprocess for both image and video processor

a few notes:

we do not need to accept list of list of 4d (the current code does not accept it). I removed the relevant test for list of list of 4d
I also think we do not need to accept list of 5d (the video processor should only accept video or a list of videos, and 5d is already a list of videos, I don't think passing list of 5d is needed); however, a list of 5d input works with current code so I make it work here as well (and add notes to it because it is a little bit confusing). I also would like to hear @DN6 ' and @a-r-r-o-w s opinions that if there would be a use case where someone needs to pass a list of 5d tensor/array as inputs - we can deprecate it if there is no use case for it

I refactored the image processor a little bit, too, so it is more aligned with the video processor

HuggingFaceDocBuilderDev · 2024-05-05T10:20:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/diffusers/video_processor.py

src/diffusers/image_processor.py

sayakpaul · 2024-05-06T04:52:56Z

src/diffusers/image_processor.py

+def is_valid_image(image):
+    return isinstance(image, PIL.Image.Image) or isinstance(image, (np.ndarray, torch.Tensor)) and image.ndim in (2, 3)
+
+
+def is_valid_image_imagelist(images):
+    # check if the image input is one of the supported formats for image and image list:
+    # it can be either (1) a 4d pytorch tensor or numpy array, (2) a valid image or (3) list of valid image
+    if isinstance(images, (np.ndarray, torch.Tensor)) and images.ndim == 4:
+        return True
+    elif is_valid_image(images):
+        return True
+    elif isinstance(images, list):
+        return all(is_valid_image(image) for image in images)
+    return False


Very important! Thank you.

src/diffusers/image_processor.py

sayakpaul

Thanks a bunch!

DN6 · 2024-05-06T12:26:30Z

I also think we do not need to accept list of 5d (the video processor should only accept video or a list of videos, and 5d is already a list of videos, I don't think passing list of 5d is needed); however, a list of 5d input works with current code so I make it work here as well (and add notes to it because it is a little bit confusing).

Agree that list of 5D is not needed.

DN6 · 2024-05-06T15:35:24Z

src/diffusers/video_processor.py

-        elif isinstance(video, list) and isinstance(video[0], PIL.Image.Image):
+        # video processor only accepts video or a list of videos or a batch of videos (5d array/tensors) as inputs,
+        # while we do accept a list of 5d array/tensors, we concatenate them to a single video batch
+        if isinstance(video, list) and isinstance(video[0], np.ndarray) and video[0].ndim == 5:


Think it's okay to not accept a list of 5D video. Perhaps just raise an error if a 5D list is passed here with a message asking to concatenate along the batch dimension?

I throw a warning and deprecated it just to be more safe

DN6

Nicely done 👍🏽

… feedbacks

sayakpaul

Let's ship!

sayakpaul · 2024-05-07T04:51:43Z

Thanks a lot, Yiyi!

* introduce videoprocessor. * fix quality * address yiyi's feedback * fix preprocess_video call. * video_processor -> image_processor * fix * fix more. * quality * image_processor -> video_processor * support List[List[PIL.Image.Image]] * change to video_processor. * documentation * Apply suggestions from code review * changes * remove print. * refactor video processor (part # 7776) (#7861) * update * update remove deprecate * Update src/diffusers/video_processor.py * update * Apply suggestions from code review * deprecate list of 5d for video and list of 4d for image + apply other feedbacks * up --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * add doc. * tensor2vid -> postprocess_video. * refactor preprocess with preprocess_video * set default values. * empty commit * more refactoring of prepare_latents in animatediff vid2vid * checking documentation * remove documentation for now. * fix animatediff sdxl * fix test failure [part of video processor PR] (#7905) up * remove preceed_with_frames. * doc * fix * fix * remove video input as a single-frame video. --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>

update

7ca8759

update remove deprecate

feb561f

yiyixuxu commented May 5, 2024

View reviewed changes

src/diffusers/video_processor.py Outdated Show resolved Hide resolved

yiyixuxu added 2 commits May 5, 2024 08:29

Update src/diffusers/video_processor.py

2721d9c

update

964508b

yiyixuxu commented May 5, 2024

View reviewed changes

src/diffusers/video_processor.py Outdated Show resolved Hide resolved

yiyixuxu commented May 5, 2024

View reviewed changes

src/diffusers/image_processor.py Outdated Show resolved Hide resolved

Apply suggestions from code review

d59a596

yiyixuxu requested a review from sayakpaul May 5, 2024 18:41

Merge branch 'video-processor' into video-processor-yiyi-testing

7a09ea3

sayakpaul reviewed May 6, 2024

View reviewed changes

src/diffusers/image_processor.py Show resolved Hide resolved

sayakpaul reviewed May 6, 2024

View reviewed changes

src/diffusers/image_processor.py Outdated Show resolved Hide resolved

sayakpaul approved these changes May 6, 2024

View reviewed changes

sayakpaul mentioned this pull request May 6, 2024

[Core] introduce videoprocessor. #7776

Merged

2 tasks

DN6 reviewed May 6, 2024

View reviewed changes

DN6 approved these changes May 6, 2024

View reviewed changes

yiyixuxu added 3 commits May 6, 2024 20:07

deprecate list of 5d for video and list of 4d for image + apply other…

90fac4f

… feedbacks

merge

3967cca

up

02594f2

sayakpaul approved these changes May 7, 2024

View reviewed changes

sayakpaul merged commit e2f61d5 into video-processor May 7, 2024

sayakpaul deleted the video-processor-yiyi-testing branch May 7, 2024 04:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor video processor (part # 7776) #7861

refactor video processor (part # 7776) #7861

Uh oh!

yiyixuxu commented May 5, 2024 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented May 5, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul May 6, 2024

Uh oh!

Uh oh!

sayakpaul left a comment

Uh oh!

DN6 commented May 6, 2024

Uh oh!

DN6 May 6, 2024 •

edited

Loading

Uh oh!

yiyixuxu May 6, 2024

Uh oh!

DN6 left a comment

Uh oh!

sayakpaul left a comment

Uh oh!

sayakpaul commented May 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

refactor video processor (part # 7776) #7861

refactor video processor (part # 7776) #7861

Uh oh!

Conversation

yiyixuxu commented May 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented May 5, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul May 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

DN6 commented May 6, 2024

Uh oh!

DN6 May 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu May 6, 2024

Choose a reason for hiding this comment

Uh oh!

DN6 left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented May 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yiyixuxu commented May 5, 2024 •

edited

Loading

DN6 May 6, 2024 •

edited

Loading