[fbgemm] check min<=max in ChooseQuantizationParams (pytorch#95529)

Summary: Pull Request resolved: pytorch#95529 X-link: pytorch/FBGEMM#1612 pytorch/FBGEMM#1590 Test Plan: CI Reviewed By: summerdengfb Differential Revision: D43383474 fbshipit-source-id: 337c82fa89b14b3633a2d61f233c98c1994bc71e
jspark1105 · Feb 26, 2023 · e088411 · e088411
1 parent 6c30dc6
commit e088411
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/caffe2/quantization/server/conv_dnnlowp_op.cc b/caffe2/quantization/server/conv_dnnlowp_op.cc
@@ -905,6 +905,10 @@ void ConvDNNLowPOp<T, ReluFused>::RunOnDeviceEpilogueNHWC_(
       Y_min = std::max(0, Y_min);
       Y_max = std::max(0, Y_max);
     }
+    if (Y_min == numeric_limits<int32_t>::max() && Y_max == numeric_limits<int32_t>::min()) {
+      Y_min = 0;
+      Y_max = 0;
+    }
 
     float Y_scale =
         in_qparams_[INPUT].scale * FilterQuantizationParams(0).scale;