"sequential model offloading" Error

[Feniksss]

Describe the bug

Hi all!
Example code:
pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
revision="fp16",
torch_dtype=torch.float16,
use_auth_token=access_token,
device_map="auto",
)

And we started getting an error

ValueError: It seems like you have activated sequential model offloading by calling enable_sequential_cpu_offload, but are now attempting to move the pipeline to GPU. This is not compatible with offloading. Please, move your pipeline .to('cpu') or consider removing the move altogether if you use sequential offloading.

The cause of error - device_map=auto
But I some confused - why didn't we get the error earlier? what has been changed?

Reproduction

pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
revision="fp16",
torch_dtype=torch.float16,
use_auth_token=access_token,
device_map="auto",
)

Logs

No response

System Info

GKE T4 cluster
torch 1.13.1+cu117
diffusers 0.14.0

[patrickvonplaten]

Hey @Feniksss,

You cannot use device_map="auto" with sequential model offloading. Please remove device_map="auto" :-)

[Feniksss]

@patrickvonplaten thanks for response! sorry, I understand that we need to remove device_map attribute. But I don't understand the reason... can you explain or give me a link to understand what has changed in the transformers? Because this is old model...

[patrickvonplaten]

Sure! The reason is that device_map="auto" will load the model directly on GPU which then means that all weights are moved to GPU. Now if you want to apply sequential model offloading the weights should to begin with not be on GPU, but on CPU - otherwise they are not offloaded.

To fix it we should start with "only CPU" as a starting position

[larme]

Hi @patrickvonplaten ,

Some of our old codes meet the same problem. What makes things a little bit complicated is that we don't call any code to enable sequential model offloading. What we do is something like this:

pipeline = StableDiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16, device_map="auto")
pipeline = pipeline.to("cuda")

Then the same error will raise. While the following code will work

pipeline = StableDiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16, device_map=None)
pipeline = pipeline.to("cuda")
pipeline = pipeline.to("cuda")
pipeline = pipeline.to("cuda")

So a pipeline loaded with device_map="auto" cannot be put on gpu again while a pipeline loaded with device=None can be put on gpu many times without any trouble.

After some tests, my understanding is that when loading with device_map="auto", some components like unet is already on GPU and is enabled with sequential model offloading, so it's needless to move the pipeline to GPU. However the error messages make me think I somehow enabled sequential model offloading myself and now cannot move the pipeline to gpu. This is where confusion happens. I wonder if we can document this better.

Another thing I want to ask is that, what's the recommending way to load a pipeline with possible gpu utilization now? Should user stick with devcie_map=auto or should they use the old way of loading the pipeline to cpu (with device_map=None) and them move the pipeline to gpu use `pipeline.to("cuda")?

[patrickvonplaten]

Interesting! I'll try to look into this!

[patrickvonplaten]

@williamberman is this related to the problem you mentioned last time we spoke?

[williamberman]

I don't think this is related to that. I would say this looks like expected behavior though. When using any feature that takes control of weight placement (device maps or model offloading), the user shouldn't be subsequently allowed to move all of the weights to the gpu. In an ideal world they would be able to but device maps and model offloading are stateful changes to the models.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.