[Question] desirable_weight and undesirable_weight in KTOTrainer

[seanexp]

I found that KTOTrainer logs warning message if desirable_weight or undesirable_weight are not in ideal bounds.

However, KTOTrainer makes balanced dataset via interleave_datasets.

Given that the training dataset is balanced, the lower_bound and upper_bound calculation seems a bit awkward.

To my understanding the equation below

des_weight_lower_bound = round((len(undesirable) * self.undesirable_weight / len(desirable)) * 1, 2)

should be rewritten into

des_weight_lower_bound = round(self.undesirable_weight, 2)

as the ratio of len(undesirable) and len(desirable) equals 1 in interleaved_train_dataset.

Please correct me if I'm wrong. (cc. @kashif @kawine )
Thanks in advance!

[kawine]

Thanks for pointing this out @seanexp !

This looks like a bug, since I was under the (incorrect) impression that interleaving datasets with all_exhausted would preserve the relative size of the original datasets, which on second look, it appears to not do. Instead, it duplicates the data in an unpredictable way, which we don't want.

The interleaving is also unnecessary, at least in the current version of the code, so I'll make a PR to remove it.

[seanexp]

Thanks! @kawine

[younesbelkada]

Hi ! This should be fixed by #1499 I believe, do you think we can close this issue?

[seanexp]

@younesbelkada Yes, I'll close this issue