Ablation studies

[karims]

I am trying to do ablation studies on Tenis dataset, different from what is done in paper for BAIR.

It looks switching off G.S is straightforward from yml file. However, switching off vt - action variability embedding and
L_act: training with the mutual information loss doesn't look that simple.

Can you shed some light on this how to proceed?

I see that to switch off L_act, there are many places to comment code. Or is it ok to set action_mutual_information_lambda and action_mutual_information_lambda_pretraining to 0? Does this work?

About v_t, I am just unable to figure that out how to switch it off in code. From the paper, it is defined as the difference between the observed action direction dt and its assigned cluster centroid. The only clue I find is in model.py line 188 says:

if not self.config["model"]["action_network"]["use_variations"]:
            flat_action_variations = flat_action_variations * 0

Does use_variations=False helps to do this ablation study?

[willi-menapace]

Dear Karims,
it is correct to proceed as you propose to deactivate L_act and v_t.

For the former, you can set action_mutual_information_lambda and action_mutual_information_lambda_pretraining to 0 in the configuration files.
For the latter, you can specify use_variations: False line in the action_network section of the configuration file.

No changes to the code should be required to reproduce the ablation.

[karims]

Thanks so much. You have done amazing work in this research.