Describe the bug
run deepspeed.init_inference and got TypeError: argument of type 'ABCMeta' is not iterable
To Reproduce
Steps to reproduce the behavior:
from transformers import AutoModelForCausalLM
import torch
model_name = 'EleutherAI/gpt-neo-2.7B'
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.half).eval()
deepspeed.init_inference(model, dtype=torch.half, replace_method='auto')
output:
[2021-11-25 02:46:40,275] [INFO] [logging.py:69:log_dist] [Rank -1] DeepSpeed info: version=0.5.8+unknown, git-hash=unknown, git-branch=unknown
[2021-11-25 02:46:40,275] [INFO] [engine.py:127:_init_quantization_setting] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Process stream_worker:
Traceback (most recent call last):
File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda/lib/python3.9/site-packages/service_streamer/service_streamer.py", line 327, in run_forever
self._model.init_model(*self._model_init_args, **self._model_init_kwargs)
File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 150, in init_model
model, tokenizer, pipeline = load_model_on_device(self.gpu_id)
File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 127, in load_model_on_device
deepspeed.init_inference(
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/__init__.py", line 274, in init_inference
engine = InferenceEngine(model,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 86, in __init__
self._apply_injection_policy(
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 161, in _apply_injection_policy
replace_transformer_layer(client_module,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 464, in replace_transformer_layer
return replace_module(model=model,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_module
replaced_module, _ = _replace_module(model, policy)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 578, in _replace_module
policies[child.__class__][0](child,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 460, in replace_fn
new_module = replace_wo_policy(child, _policy)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 443, in replace_wo_policy
return _replace_module(module)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
_replace_module(child, name)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
_replace_module(child, name)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 435, in _replace_module
linear_policies[child.__class__](child,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 344, in _replace
if name in all_reduce_linears:
TypeError: argument of type 'ABCMeta' is not iterableExpected behavior
A clear and concise description of what you expected to happen.
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
async_io ............... [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/miniconda/lib/python3.9/site-packages/torch']
torch version .................... 1.10.0+cu113
torch cuda version ............... 11.3
nvcc version ..................... 11.0
deepspeed install path ........... ['/root/miniconda/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.5.8+unknown, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.10, cuda 11.3
System info (please complete the following information):
- OS: Ubuntu 18.04
- GPU count and types [one machines with x4 V100]
- Python version: 3.9
- Any other relevant info about your setup: I also tried compiled from source
Describe the bug
run
deepspeed.init_inferenceand gotTypeError: argument of type 'ABCMeta' is not iterableTo Reproduce
Steps to reproduce the behavior:
output:
Expected behavior
A clear and concise description of what you expected to happen.
ds_report output
System info (please complete the following information):