Investigate Validation Set

[JoshVarty]

Currently our train set overperforms the test set. Why is this? How can we improve?

[JoshVarty]

Train

• 4,970 clips from Freesound
• 19,815 clips from Flickr

Test

• 1,120 from Freesound

This suggests to me that we should:

1. Choose validation examples from Freesound
2. Ensure the classes are balanced? What does the balance look like anyway?

[JoshVarty]

So we've created it, now we need to test the validation scores against the test scores. Are they close? Does increasing validation score also increase test score?

[JoshVarty]

Taken from #25

Curated Training Set

Noisy Training Set

Almost all clips are 15 seconds long.

Test Set

So it looks to me like the test set is taken from roughly the same distribution as the curated training set.

[JoshVarty]

I'm feeling pretty good about the results of our validation lwlwrap compared to our test lwlwrap. Whenever we see an increase in validation score, we get an increase in LB score. See #26

The gap between our validation score and test score may be due to:

1. The test clips are slightly longer. (Fewer clips are less than 2 seconds long)
2. The leaderboard score may not be calculated across all results, perhaps to make it harder to game?