-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
some error in the traindata #43
Comments
It seems you are getting NaN. You may need to investigate further into your model, data loader, training parameters etc. It is not easy to find culprit in these cases. I would try plotting the gradients over the course of the training. |
@erkil1452 I use the data in your train.txt, and the model predict the gaze using only single RGB image. so my training data is only within the train.txt, each line in the train.txt is a training data. |
I am noticing you get NaN only for the angular error. If this is the source of your issues I suspect that you have not clamped the input values you pass into a |
the validate log is: the model in that status is the best? the validate angular error is still large. Is the model overfitting? what suggestion do you give when the model overfitting? |
The NaN would completely spoil the network weights and make the model unusable. If your model is overfitting you can either try to enlarge your training set, add augmentation to your training data, add noise, Dropouts, ... or just stop the training early. |
@erkil1452 I trained a model using your train.txt, but there are some warning, I guess there are some error in you training data:
Epoch: [325][651/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.076 (2.421) Loss 0.0120 (0.0128) Prediction Error 2.5859 (2.5833)
Epoch: [325][652/1587] Time 0.722 (0.724) Data 0.009 (0.013) Angular 2.126 (2.421) Loss 0.0104 (0.0128) Prediction Error 2.4450 (2.5831)
Epoch: [325][653/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.402 (2.421) Loss 0.0133 (0.0128) Prediction Error 2.4985 (2.5830)
WARNING:root:NaN or Inf found in input tensor.
Epoch: [325][654/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular nan (nan) Loss 0.0152 (0.0128) Prediction Error 2.5007 (2.5829)
Epoch: [325][655/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.482 (nan) Loss 0.0134 (0.0128) Prediction Error 2.5017 (2.5828)
Epoch: [325][656/1587] Time 0.720 (0.724) Data 0.010 (0.012) Angular 2.701 (nan) Loss 0.0138 (0.0128) Prediction Error 2.8752 (2.5832)
Epoch: [325][657/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 1.995 (nan) Loss 0.0106 (0.0128) Prediction Error 2.3353 (2.5828)
Epoch: [325][658/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.179 (nan) Loss 0.0112 (0.0128) Prediction Error 2.4611 (2.5826)
Epoch: [325][659/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.341 (nan) Loss 0.0140 (0.0128) Prediction Error 2.5880 (2.5826)
Epoch: [325][660/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 2.273 (nan) Loss 0.0114 (0.0128) Prediction Error 2.2808 (2.5822)
Epoch: [325][661/1587] Time 0.720 (0.724) Data 0.010
The text was updated successfully, but these errors were encountered: