txt2img-example.py failed with unexpected keyword argument 'encoder_hidden_states'

[wenkechen]

https://github.com/microsoft/DeepSpeed-MII/blob/v0.0.4/examples/local/txt2img-example.py

The following are error message.

Traceback (most recent call last):
File "/root/moab/code/DeepSpeed-MII/examples/local/txt2img-example.py", line 29, in
result = generator.query({
File "/root/moab/code/DeepSpeed-MII/mii/server_client.py", line 367, in query
response = self.asyncio_loop.run_until_complete(
File "/opt/conda/envs/mii/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/root/moab/code/DeepSpeed-MII/mii/server_client.py", line 263, in _query_in_tensor_parallel
await responses[0]
File "/root/moab/code/DeepSpeed-MII/mii/server_client.py", line 313, in _request_async_response
response = await self.stubs[stub_id].Txt2ImgReply(req)
File "/opt/conda/envs/mii/lib/python3.9/site-packages/grpc/aio/_call.py", line 290, in await
raise _create_rpc_error(self._cython_call._initial_metadata,
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Exception calling application: forward() got an unexpected keyword argument 'encoder_hidden_states'"
debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:50050 {created_time:"2023-01-12T07:39:05.68597804+00:00", grpc_status:2, grpc_message:"Exception calling application: forward() got an unexpected keyword argument 'encoder_hidden_states'"}"

Setup

transformers==4.25.1
diffusers==0.11.1
deepspeed==0.7.7
deepspeed-mii==0.0.4

[mrwyattii]

Hi @chenwaynepaul I can confirm that I'm seeing the same error. It appears to be related to a recent change in diffusers. I'm still debugging the issue, but to get you unstuck use diffusers==0.10.2 and it should work.

[mrwyattii]

A forward method signature changed in the latest diffusers. Here is a PR on DeepSpeed that should fix the issue. Can you please try this and report back? Thanks!

microsoft/DeepSpeed#2699

edit: This PR is now part of v0.8.0 of DeepSpeed. Please update to the latest version and try again.

[mrwyattii]

Closing, please reopen if this did not solve your problem.

[wenkechen]

Cool! Thanks @mrwyattii