Unable to run mii-sd.py for txt2img benchmark

[aponte411]

https://github.com/microsoft/DeepSpeed-MII/tree/main/examples/benchmark/txt2img

Ive run into some strange protobuf related errors. When I first ran into this, I was able to resolve by changing my protobuf version to >=3.20.0 but now it doesnt work anymore.

My hunch is that its related to how I am installing things? I wasnt sure what the correct way was to install deespeed and deepspeed-mii, so I have been trying to use the following:
pip install deepspeed[sd] deepspeed-mii

I am now seeing this error when trying to run mii-sd.py:

Setup:
deepspeed==0.7.6
deepspeed-mii==0.0.4
diffusers==0.9.0
Ubuntu 18.0.4.6 LTS
py3.8.15

[mrwyattii]

What's the version of protobuf you have in this environment? pip list | grep proto

Also, you can avoid this by setting the following environment variable (but as the error message notes, it could add latency to the message passing): export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python.

[aponte411]

@mrwyattii when I install using pip install deepspeed[sd] deepspeed-mii it installs protobuf==4.21.11. Using this version, I ran into the TypeError: Descriptors cannot be created directly. I tried downgrading but that may have messed it up. I tried again setting that env variable, but keeping this version 4.21.11, and that seems to have resolved the protobuf issue but now the AttributeError from this issue seems to pop back up.

I checked and I am using deepspeed==0.7.6 and deepspeed-mii==0.0.4. Am I missing something?

[mrwyattii]

I think you're doing things correct. Let me try to follow these steps in a fresh environment and see if I can reproduce.

[mrwyattii]

@aponte411 I'm not able to reproduce the protobuf error installing with a clean environment. pip installs protobuf==3.20.3 for me. But regarding the other issue after you have set the environment variable... What version of transformers do you have? I just checked and it looks like the latest transformers version is breaking our SD example. I would recommend version 4.24.0 until we can find a solution.

[aponte411]

Looks like the protobuf issue was due to my local setup. Creating a new venv with a fresh install fixes it! @mrwyattii I'm using transformers==4.24.0 but I am still seeing the DSUNet config issue.

[mrwyattii]

@aponte411 try this PR on the DeepSpeed side, should fix the config problem you are seeing:
microsoft/DeepSpeed#2589

[aponte411]

@mrwyattii works when installing from master :)