Errors running Zero-Inference text generation example

[joaopcm1996]

Hey,

I'm trying to run the example provided for text generation with Zero-Inference, and having trouble getting predictions without running into errors.

When I try to deploy the exact same model and config, I first get a validation error for the aio configuration.

Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/mii/launch/multi_gpu_server.py", line 70, in <module>
    main()
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/mii/launch/multi_gpu_server.py", line 56, in main
    inference_pipeline = load_models(task_name=args.task_name,
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/mii/models/load_models.py", line 87, in load_models
    ds_config = DeepSpeedConfig(ds_config_path)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/deepspeed/runtime/config.py", line 811, in __init__
    self._initialize_params(copy.copy(self._param_dict))
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/deepspeed/runtime/config.py", line 830, in _initialize_params
    self.zero_config = get_zero_config(param_dict)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/deepspeed/runtime/zero/config.py", line 66, in get_zero_config
    return DeepSpeedZeroConfig(**zero_config_dict)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/deepspeed/runtime/config_utils.py", line 54, in __init__
    super().__init__(**data)
  File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for DeepSpeedZeroConfig
aio
  extra fields not permitted (type=value_error.extra)

If I remove the aio config the server starts successfully, but as I'm trying to create a generator and query it (just like in your sample, I get another error for the generator.query() call:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_5004/1158427391.py in <cell line: 1>()
----> 1 result = generator.query({'query': ["DeepSpeed is the", "Seattle is"]})

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/mii/server_client.py in query(self, request_dict, **query_kwargs)
    357         else:
    358             assert self.initialize_grpc_client, "grpc client has not been setup when this model was created"
--> 359             response = self.asyncio_loop.run_until_complete(
    360                 self._query_in_tensor_parallel(request_dict,
    361                                                query_kwargs))

~/anaconda3/envs/pytorch_p38/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    590         """
    591         self._check_closed()
--> 592         self._check_running()
    593 
    594         new_task = not futures.isfuture(future)

~/anaconda3/envs/pytorch_p38/lib/python3.8/asyncio/base_events.py in _check_running(self)
    550     def _check_running(self):
    551         if self.is_running():
--> 552             raise RuntimeError('This event loop is already running')
    553         if events._get_running_loop() is not None:
    554             raise RuntimeError(

RuntimeError: This event loop is already running

Any help is greatly appreciated.

[mrwyattii]

Hi @joaopcm1996 and thanks for trying out MII. Looks like we had a bug in the DeepSpeed config for text-generation-zero-example.py. I've fixed that in #88

With the updated script, I was able to run the example with:

python3 text-generation-zero-example.py
python3 text-generation-query-example.py -d "gpt2-xl_deployment"

The error you see when querying may be due to another MII server running in the background, but I'm not certain. You can shut down a server with:

from mii import terminate
terminate("gpt2-xl_deployment") #or replace with whatever the deployment name is

[jeffra]

@joaopcm1996, is this now resolved on your side? Please reopen if you still have an issue here.

[joaopcm1996]

I'll test it out with your recommendation and get back to you if I run into any issues. Appreciate the help!