minillm: Run DeepSpeed. After initialization, some parameters of the model are missing, such as model_ori.transformer.h[0].attn.c_attn.wight.

[liuxy1103]

def setup_model_and_optimizer(args, ds_config, device, set_optim=True):
# get the model
model = get_model(args, device)
# get the optimizer and lr_scheduler
if set_optim:
optimizer = get_optimizer(args, model)
lr_scheduler = get_learning_rate_scheduler(args, optimizer)
else:
optimizer, lr_scheduler = None, None
model_ori = model
model, optimizer, _, lr_scheduler = deepspeed.initialize(
model=model,
optimizer=optimizer,
args=args,
lr_scheduler=lr_scheduler,
mpu=mpu if args.model_parallel else None,
config_params=ds_config

    Run DeepSpeed. After initialization, some parameters of the model are missing, such as model_ori.transformer.h[0].attn.c_attn.wight.

[t1101675]

Hi, can you provide more detailed information, like the running command, the scripts, and the environment?

[liuxy1103]

I modify the setting in minillm/configs/deepspeed/ds_config.json
"zero_optimization": {
"stage": 3,
"offload_param":{
"device": "cpu"
}
},

[t1101675]

We currently do not support zero3 and cpu-offload. Maybe we will add this feature in a later version.