Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Valid bug fix #25

Merged
merged 6 commits into from Jul 14, 2021
Merged

Valid bug fix #25

merged 6 commits into from Jul 14, 2021

Conversation

ws-choi
Copy link
Collaborator

@ws-choi ws-choi commented Jul 14, 2021

  • self.log in validation step with sync_dist=True is not enough to log the exact validation loss for musdb18 especially if you are using multi gpus.

Environment

  • 4 gpus ddp
  • 14 tracks in validation step
  • 1 batched dataloader

outcomes

  • 16 tracks not 14 tracks are evaluated

  • why?

    • node1: 0 1 2 3
    • node2: 4 5 6 7
    • node3: 8 9 10 11
    • node4: 12 13 0 1 <= bolds are not supposed to be here
  • It seems a kind of bug of pytorch-lightning.

  • using drop_last=True also does not work for this issue.

  • I updated ugly code to fix this.

@ws-choi ws-choi added this to In progress in KUIELAB's MDX-Net via automation Jul 14, 2021
@ws-choi ws-choi linked an issue Jul 14, 2021 that may be closed by this pull request
@ws-choi ws-choi merged commit f069112 into main Jul 14, 2021
KUIELAB's MDX-Net automation moved this from In progress to Done Jul 14, 2021
@ws-choi ws-choi deleted the valid_bug_fix branch July 14, 2021 04:37

def __getitem__(self, index):
if index >= self.num_true_tracks:
return -1, -1, -1, -1, -1
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolved bug by adding dummy instances


return {'track_id': index, 'track': target_wav_hat}
if num_tracks < 0:
self.log("val/sdr", 0, prog_bar=False, logger=True, on_step=False, on_epoch=True,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve bug by adding dummy instances

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Validation for DDP
1 participant