how to get the parameters of model after using ZeroInitContext to init zero

[Lyn-Lucy]

from colossalai.zero.init_ctx import ZeroInitContext

with ZeroInitContext(target_device=torch.cuda.current_device(),
                    shard_strategy=shard_strategy,
                    shard_param=True):
        model = AlbertForClozeTest(args.model_name_or_path).to(args.device)
        print(list(model.parameters()))

here is the result

tensor([], device='cuda:3', dtype=torch.float16, requires_grad=True), Parameter containing:
tensor([], device='cuda:3', dtype=torch.float16, requires_grad=True), Parameter containing:

and here is my config.py of zero

from colossalai.zero.shard_utils import TensorShardStrategy

from colossalai.amp import AMP_TYPE
    
# fp16 = dict(
#   mode=AMP_TYPE.NAIVE
# )
# gradient_accumulation = 16
# CONFIG = dict(fp16=dict(mode=AMP_TYPE.TORCH),
zero = dict(
    model_config=dict(
        shard_strategy=TensorShardStrategy(),
        reduce_scatter_bucket_size_mb=25,
        fp32_reduce_scatter=False,
        tensor_placement_policy="cuda",
        gradient_predivide_factor=1.0,
        reuse_fp16_shard=False
    ),
    optimizer_config=dict(
        gpu_margin_mem_ratio=0.8,
        initial_scale=2**5,
        min_scale=1,
        growth_factor=2,
        backoff_factor=0.5,
        growth_interval=1000,
        hysteresis=2,
        max_scale=2**32
    )
)

I can't get the parameters of my model

Until now, I can't figure out the problem, so I need a favor. Please give me some advice or some similar working examples, thanks.

[feifeibear]

https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/gpt/train_gpt_demo.py
hello, can you refer this example to use ZeRO.
The ZeroInitContext is an old API, we will not maintain it anymore.

[Lyn-Lucy]

thank you for your answer. I change the way to init Zero but there is another error.

here is the example I refer

ColossalAI-Examples/train_v2.py at main · hpcaitech/ColossalAI-Examples (github.com)

and the error is

RuntimeError: Found dtype Half but expected Float

and I have another question,should I use my config file of colossalai in this place

colossalai.launch_from_torch(config={})

I would appreciate it if you could help me solve these questions!

[Lyn-Lucy]

After I use the optimizer in the example ,it works!

optimizer = HybridAdam(model.parameters(), lr=1e-3)
optimizer = ZeroOptimizer(optimizer, model, initial_scale=2**5)