Fix random validation errors for good (and restore torch.compile for the validation pipeline at the same time)

[mhirki]

I saw the random validation errors were back after commit 48cfc09 removed some of my earlier fixes for the random validation issues. This time I decided to dig to the bottom of the issue of why the pipeline is randomly on the cpu.

The root cause turned out to be an early return in setup_pipeline which means that this line at the end of setup_pipeline is never executed:

        self.pipeline = self.pipeline.to(self.inference_device)

And torch.compile before that is also never called.

I did notice that moving the entire pipeline to the accelerator device has some unwanted side-effects. It also moved text_encoder_2 and text_encoder_3 to the GPU when I was testing with SD3.5. Setting text_encoder_2 and text_encoder_3 to None in extra_pipeline_kwargs prevents them from being loaded in the first place. I'm hoping other pipelines won't complain about text_encoder_2 and text_encoder_3 in kwargs. I tried throwing in text_encoder_4 there and the SD3 pipeline didn't complain.

I tried testing validation_torch_compile but it seemed to be taking forever so I eventually just ended up killing it.

[bghira]

the reason this is done is because some of diffusers pipelines require the text encoder to be there simply so that its dtype can be checked 😮‍💨

i could copy those pipelines in and fix them, but they'll need to be checked for compatibility fully first.

i don't want to move the whole pipeline to GPU when doing validations because the necessary components are already there - the text encoders then remained on CPU. but it'd be better for them to not be loaded at all.