New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Should ONNX have Dropout / request for FeatureDropout #113
Comments
Although dropout is nop in inference, but they are still in the model saved after training. Also, no one prevent anyone from loading the model in CNTK or PyTorch or Caffe2 and do fine tuning. When you load ONNX in CNTK, it just covert ONNX model to CNTK internal presentation, in theory you can continue training, replace some nodes...etc. Regarding FeatureDropout, it isn't clear to me from the code you share what it does? Can you provide more detail? |
My bad, I should have motivated this better (and also, perhaps FeatureDropout is not a very good name). "FeatureDropout", also known as "Dropout2d/Dropout3d" in PyTorch (http://pytorch.org/docs/master/nn.html#torch.nn.Dropout2d) slightly modifies the Dropout algorithm so that only channels (the first two dimensions as in NCHW ordering; the important thing is that it matches the output of Conv) are randomly masked out; the actual pixels aren't randomly masked (since the correlations between adjacent pixels means that the masking is just decreasing the effective learning rate.) It's a fairly important technique and shows up in some models that we are interested in exporting. |
Instead of coming up with names for additional dropout variants, consider adding a parameter to the existing Dropout that gives the list of axes to tie together (i.e., an empty list by default, [0] for batchwise dropout, [1] for dropping pixels/voxels in all channels, [2,3] for 2d spatial dropout, [2,3,4] for 3d spatial dropout, all assuming NC012... layout). This keeps the namespace simple and gives much greater flexibility in what can be represented. |
* Fix compilation of getSupportedAPITest by removing old header * Added parseFromFile function and updated createParser function from TRT 5.0 NvOnnxParser headers. * Merging from master * Fix typo * WIP fixing backend tests * Fix comments * Undoing WIP on backend * Revert to ParseFromFile_WAR for getSupportedApiTest
* Add Dropout Based on the implementation in #51 Unlike PR #51, I don't use "feature dropout" like nn.Dropout2d or F.dropout2d, as ONNX does not define them [1]. [1] onnx/onnx#113 * Remove unneeded error bounds for tests * Use self.training from PyTorch instead one from ONNX
If ONNX is intended to be an inference oriented framework, arguably it should not have Dropout at all (since this is a purely training-time construct).
If we decide we should keep Dropout, then I'd also like to request adding FeatureDropout to ONNX https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/dropout.py#L58
The text was updated successfully, but these errors were encountered: