-a/--ablation flag not working as expected

[mulhod]

I am using run_experiment with the -a flag and passing in a value of 3.

run_experiment -a 3 experiments/xval/MH_old_again/configs/min+max+lpmc+lpmm.cfg &

In the config file, there is one feature-set that contains four feature file IDs.

featuresets = [["min", "max", "lpmc", "lpmm"]]

I would like to do an ablation experiment where a system is trained with features in one of those feature files at a time. However, while running run_experiment with a debugger, I noticed that the "featuresets" variable (this is in experiments.py) evaluates to:

>>> featuresets
[['min'], ['max'], ['lpmm'], ['lpmc'], ['lpmc', 'lpmm', 'max', 'min']]

This shows that each feature-file ID is included as I expected, but that there's also an extra set that includes all of the feature files. As I understand it, this should result in [['min'], ['max'], ['lpmm'], ['lpmc']].

Is this incorrect?

[desilinguist]

No. All ablation experiments include a run with all the features as a baseline.

[mulhod]

In the readthedocs tutorial page under the Using run_experiment section part for the ablation flag:

Runs an ablation study where repeated experiments are conducted with the specified number of feature files in each featureset in the configuration file held out. For example, if you have three feature files (A, B, and C) in your featureset and you specifiy --ablation 1, there will be three three experiments conducted with the following featuresets: [[A, B], [B, C], [A, C]].

Should the featuresets include [A, B, C], then?

[desilinguist]

Yeah, I guess so. Can you please submit a pull request?

[mulhod]

Submitted pull request.

[dan-blanchard]

Fixed in #226.