[REQUEST] GAN: Different learning rates and schedulers in one config for multiple .initialize() calls possible?

[fteufel]

Hi, I'm using deepspeed to train a model that has a discriminator. I'm following the official GAN tutorial mostly (https://www.deepspeed.ai/tutorials/gan/)

Essentially I set up the model like this

model_engine, optimizer, _, _ = deepspeed.initialize(args=args, model = model, model_parameters=model.parameters(), optimizer=optimizer)
model_engine_adversary, optimizer_adversary, _, _ = deepspeed.initialize(args=args, model = adversarial_head, model_parameters=adversarial_head.parameters(), optimizer=optimizer_adversary)

And then start it with

deepspeed train.py  --deepspeed_config deepspeed_config.json

If I specify a scheduler in the json config as done in all tutorials, it applies to both model engines. Is there a way to specify two separate ones?

[tjruwase]

You can pass a dict using the config argument to deepspeed.initialize() just like here. This way you can pass separate dicts to the engines.