How do I ensure no data leakage in the validation split

[aribornstein]

What is your question?

I'm working on a dataset that has some identifiable information which would lead to data leakage if not the data is not split properly. Currently the validation split is hard coded in each respective DataModules.

        if valid_split:
            full_length = len(train_ds)
            train_split = int((1.0 - valid_split) * full_length)
            valid_split = full_length - train_split
            train_ds, valid_ds = torch.utils.data.random_split(
                train_ds,
                [train_split, valid_split],
                generator=torch.Generator().manual_seed(seed)
            )

Ideally I'd like a flag that enables me to ensure there is no overlap in the train and validation data on these fields by rebalancing any overlap. Due to the way dataset initialization is hardcoded once the validation dataset is created it becomes immutable.

What is the best way to handle such a check, in Flash?