[HacktoberFest] make fast test for `StableDiffusionXLPipeline` smaller when applicable

[sayakpaul]

This feature request is included in this year's Diffusers Hacktoberfest 🎃!

Hacktoberfest's purpose is to give upcoming open-source developers the opportunity to learn about open-source and programming by fixing simple issues (like this one).

How to solve the issue:

• 1. Every issue has a description further below explaining the problem. First, you should try to understand the issue.
• 2. Second, follow the contribution guide to learn how to contribute to Diffusers.\
• 3. Claim this issue for you by opening a PR that links to this issue and writing, "I am solving this issue in <link-to-pr>".

Issue

We want to run our fast tests run faster. But what is a fast test for a pipeline, though?

This class holds all fast tests here:

diffusers/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl.py

Line 44 in bbe8d3a

class StableDiffusionXLPipelineFastTests(PipelineLatentTesterMixin, PipelineTesterMixin, unittest.TestCase):

The idea of this issue is to try to identify if the pipeline components (such as the UNet, text encoders, VAE) could be made smaller without disturbing the correctness of the tests.

For example, could we reduce the number of blocks in the UNet? This is how the UNet is initialized:

diffusers/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl.py

Line 53 in bbe8d3a

unet = UNet2DConditionModel(

Similar idea could be applied to the VAE, and text encoders (one, two).

As a consequence of making the pipeline components smaller, the expected assertion values (example) might change but it should not impact the correctness of the test, whatsoever.

Cc: @DN6