Use test dataset as validation set

[minwang-ai]

Hi Yisheng,

How did you validate the model and tune hyper parameter?
I found that you used test dataset as validation set and also used test set for testing, i.e., you use test data during training.

I learned from Machine learning course that a learning curve of loss on training and validation datasets will indicate overfitting if the curve of training data drops and may plateau and the curve for validation set will drops at first and then begins to rise.

It's important to use new data when evaluating our model to prevent the likelihood of overfitting to the training set. However, sometimes it's useful to evaluate our model as we're building it to find that best parameters of a model - but we can't use the test set for this evaluation or else we'll end up selecting the parameters that perform best on the test data but maybe not the parameters that generalize best. To evaluate the model while still building and tuning the model, we create a third subset of the data known as the validation set. A typical train/test/validation split would be to use 60% of the data for training, 20% of the data for validation, and 20% of the data for testing.

It is also important to shuffle the data before making these splits and training, however, you set shuffle to false for the training data loader.

[ethnhe]

• About the model validation and hyperparameter tuning, our strategy is following previous work like PoseCNN & DenseFusion for a fair comparison. The creator of the LineMOD and YCB Video dataset didn't split a validation set so all following work train in this way.
• About the generalization and overfitting concern, we've tried testing the selected model with a new RGBD sensor without extra training and found the model generalizes well.
• For training data shuffling, in our data loader, we get each item by random selecting, which is the same as shuffling.

[minwang-ai]

• About the model validation and hyperparameter tuning, our strategy is following previous work like PoseCNN & DenseFusion for a fair comparison. The creator of the LineMOD and YCB Video dataset didn't split a validation set so all following work train in this way.

Do you mean they all train model on training set and do hyper parameters tuning on test set and evaluate on test set
or train model without hyper parameter tuning (use same hyper parameter of previous work), since your validation set and test set are all from test.txt?

[ethnhe]

Yes, you can read their codes for more detail, for example, DenseFusion. Some of our data processing strategies are following that project.