Why does the grad-norm saturate in Stage-2 ?

[jis3613]

Hi, I'm training the stage-2 of QueST for LIBERO-90 with the autoencoder trained on LIBERO-90 at stage-0.

While monitoring the dashboard of wandb, I noticed that the grad-norm increases and saturates at 100 due to gradient clipping.
Why does this happen?
Isn't grad-norm supposed to be decreasing as the training progresses?

Thanks!

[atharvamete]

Hi, in the prior training the grad norm usually remains high throughout the training but we never observed it to go beyond 5. Which script are you running?

[jis3613]

python train.py --config-name=train_fewshot.yaml
task=libero_long
algo=quest
exp_name=final
variant_name=block_32_ds_4_LIBERO_90_stage_1_loaded
training.use_tqdm=false
training.save_all_checkpoints=true
training.use_amp=false
train_dataloader.persistent_workers=true
train_dataloader.num_workers=6
make_unique_experiment_dir=false
algo.skill_block_size=32
algo.downsample_factor=4
algo.l1_loss_scale=10
training.n_epochs=100
rollout.interval=50
rollout.num_parallel_envs=1
rollout.rollouts_per_env=1
checkpoint_path='~/QueST/experiments/libero/LIBERO_90/quest/final/block_32_ds_4/0/stage_1'
seed=0

I'm training a model on LIBERO-10 with stage-2 script with the model pre-trained on LIBERO-90 with stage-1 training.
And I observe a saturation of grad-norm at 100.

[atharvamete]

Yes, with algo.l1_loss_scale=10 the grad norm remains high as autoencoder weights are being constantly adjusted to match the sampled sequences from the prior. With algo.l1_loss_scale=0 the grad norm increases till 2.2 initially and then constantly decreases.
Moreover, the task should be task=libero_long_fewshot, with latest commit this should work without errors.

[jis3613]

Okay I see, but I observe the same phenomenon during stage-2 training when I do the following two experiments.

stage-0 on LIBERO-10 => stage-1 on LIBERO-10 => stage-2 on LIBERO-10
and
stage-0 on LIBERO-90 => stage-1 on LIBERO-90 => stage-2 on LIBERO-10

Even with the updated 'task=libero_long_fewshot' I still observe the very high grad_norm during stage 2.

I guess it is also due to the decoder L1 loss to match the sampled sequences from the prior...

Thanks for your answer.