Cherry pick commits from master to 1.9.0 release branch (#3392)

* Upgrade protobuf-compiler in Linux-CI (#3378)

* upgrade protobuf-compiler in Linux-CI

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* cd back to the rignt dir

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* sudo make

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* bug fix for proto utils and update checker error messages (#3373)

* bug fix for proto utils and update checker error messages

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

* plus add comments

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

* Updating BatchNorm outputs for training mode (#3379)

* Updating BatchNorm outputs for training mode based on the discussions with ORT team.

Signed-off-by: Negin Raoof <neginmr@utexas.edu>

* Removing outputs

Signed-off-by: Negin Raoof <neginmr@utexas.edu>
Signed-off-by: Ewa21 <ewa.tusien@intel.com>

* Bump version to 1.8.202.

Signed-off-by: Ewa21 <ewa.tusien@intel.com>

Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com>
Co-authored-by: Ashwini Khade <askhade@microsoft.com>
Co-authored-by: Negin Raoof <neginmr@utexas.edu>

.azure-pipelines/Linux-CI.yml

@@ -44,9 +44,23 @@ jobs:
       python -m pip install virtualenv
       python -m virtualenv py$(python.version)
       source py$(python.version)/bin/activate
+
+      git clone https://github.com/protocolbuffers/protobuf.git
+      cd protobuf
+      git checkout 3.11.x
+      git submodule update --init --recursive
+      mkdir build_source && cd build_source
+
+      cmake ../cmake -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
+      export NUM_PROCESSOR=`grep -c ^processor /proc/cpuinfo`
+      sudo make -j${NUM_PROCESSOR}
+      sudo make install
+      ldconfig
+      cd ../..
+
       python -m pip install --upgrade pip
-      python -m pip install numpy protobuf
-      sudo apt-get install protobuf-compiler libprotoc-dev
+      python -m pip install numpy protobuf==3.11.3
+
       sudo apt-get install -qq -o=Dpkg::Use-Pty=0 -y --no-install-recommends dos2unix
       git submodule update --init --recursive
       export ONNX_BUILD_TESTS=1

VERSION_NUMBER

@@ -1 +1 @@
-1.8.201
+1.8.202

docs/Changelog.md

@@ -18633,19 +18633,21 @@ This version of the operator has been available since version 14 of the default
   and the running statistics in training mode (training_mode=True).
   There are multiple cases for the number of outputs, which we list below:
 
-  Output case #1: Y, running_mean, running_var, current_mean, current_var (training_mode=True)
+  Output case #1: Y, running_mean, running_var (training_mode=True)
   Output case #2: Y (training_mode=False)
 
   When training_mode=False, extra outputs are invalid.
   The outputs are updated as follows when training_mode=True:
   ```
-  current_mean = ReduceMean(X, axis=all_except_channel_index)
-  current_var =  ReduceVar(X, axis=all_except_channel_index)
-
   running_mean = input_mean * momentum + current_mean * (1 - momentum)
   running_var = input_var * momentum + current_var * (1 - momentum)
 
   Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
+
+  where:
+
+  current_mean = ReduceMean(X, axis=all_except_channel_index)
+  current_var =  ReduceVar(X, axis=all_except_channel_index)
   ```
 
   When training_mode=False:
@@ -18687,7 +18689,7 @@ This version of the operator has been available since version 14 of the default
 <dd>running (training) or estimated (testing) variance tensor of shape (C).</dd>
 </dl>
 
-#### Outputs (1 - 5)
+#### Outputs (1 - 3)
 
 <dl>
 <dt><tt>Y</tt> (differentiable) : T</dt>
@@ -18696,10 +18698,6 @@ This version of the operator has been available since version 14 of the default
 <dd>The running mean after the BatchNormalization operator.</dd>
 <dt><tt>running_var</tt> (optional, non-differentiable) : T</dt>
 <dd>The running variance after the BatchNormalization operator.</dd>
-<dt><tt>current_mean</tt> (optional, non-differentiable) : T</dt>
-<dd>Current mean used during training to speed up gradient computation.</dd>
-<dt><tt>current_var</tt> (optional, non-differentiable) : T</dt>
-<dd>Current variance used during training to speed up gradient computation.</dd>
 </dl>
 
 #### Type Constraints

docs/Operators.md

@@ -1863,19 +1863,21 @@ expect(node, inputs=[x], outputs=[y], name='test_averagepool_3d_default')
   and the running statistics in training mode (training_mode=True).
   There are multiple cases for the number of outputs, which we list below:
 
-  Output case #1: Y, running_mean, running_var, current_mean, current_var (training_mode=True)
+  Output case #1: Y, running_mean, running_var (training_mode=True)
   Output case #2: Y (training_mode=False)
 
   When training_mode=False, extra outputs are invalid.
   The outputs are updated as follows when training_mode=True:
   ```
-  current_mean = ReduceMean(X, axis=all_except_channel_index)
-  current_var =  ReduceVar(X, axis=all_except_channel_index)
-
   running_mean = input_mean * momentum + current_mean * (1 - momentum)
   running_var = input_var * momentum + current_var * (1 - momentum)
 
   Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
+
+  where:
+
+  current_mean = ReduceMean(X, axis=all_except_channel_index)
+  current_var =  ReduceVar(X, axis=all_except_channel_index)
   ```
 
   When training_mode=False:
@@ -1919,7 +1921,7 @@ Other versions of this operator: <a href="Changelog.md#BatchNormalization-1">1</
 <dd>running (training) or estimated (testing) variance tensor of shape (C).</dd>
 </dl>
 
-#### Outputs (1 - 5)
+#### Outputs (1 - 3)
 
 <dl>
 <dt><tt>Y</tt> (differentiable) : T</dt>
@@ -1928,10 +1930,6 @@ Other versions of this operator: <a href="Changelog.md#BatchNormalization-1">1</
 <dd>The running mean after the BatchNormalization operator.</dd>
 <dt><tt>running_var</tt> (optional, non-differentiable) : T</dt>
 <dd>The running variance after the BatchNormalization operator.</dd>
-<dt><tt>current_mean</tt> (optional, non-differentiable) : T</dt>
-<dd>Current mean used during training to speed up gradient computation.</dd>
-<dt><tt>current_var</tt> (optional, non-differentiable) : T</dt>
-<dd>Current variance used during training to speed up gradient computation.</dd>
 </dl>
 
 #### Type Constraints
@@ -2003,18 +2001,18 @@ var = np.array([1, 1.5]).astype(np.float32)
 # using np.bool(1) while generating test data with "'bool' object has no attribute 'dtype'"
 # working around by using np.byte(1).astype(bool)
 training_mode = 1
-y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
+y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
 
 node = onnx.helper.make_node(
     'BatchNormalization',
     inputs=['x', 's', 'bias', 'mean', 'var'],
-    outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+    outputs=['y', 'output_mean', 'output_var'],
     training_mode=training_mode
 )
 
 # output size: (1, 2, 1, 3)
 expect(node, inputs=[x, s, bias, mean, var],
-       outputs=[y, output_mean, output_var, saved_mean, saved_var],
+       outputs=[y, output_mean, output_var],
        name='test_batchnorm_example_training_mode')
 
 # input size: (2, 3, 4, 5)
@@ -2026,20 +2024,20 @@ var = np.random.rand(3).astype(np.float32)
 training_mode = 1
 momentum = 0.9
 epsilon = 1e-2
-y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
-                                                                            epsilon)
+y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
+                                                      epsilon)
 
 node = onnx.helper.make_node(
     'BatchNormalization',
     inputs=['x', 's', 'bias', 'mean', 'var'],
-    outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+    outputs=['y', 'output_mean', 'output_var'],
     epsilon=epsilon,
     training_mode=training_mode
 )
 
 # output size: (2, 3, 4, 5)
 expect(node, inputs=[x, s, bias, mean, var],
-       outputs=[y, output_mean, output_var, saved_mean, saved_var],
+       outputs=[y, output_mean, output_var],
        name='test_batchnorm_epsilon_training_mode')
 ```
 

docs/TestCoverage.md

@@ -1376,18 +1376,18 @@ var = np.array([1, 1.5]).astype(np.float32)
 # using np.bool(1) while generating test data with "'bool' object has no attribute 'dtype'"
 # working around by using np.byte(1).astype(bool)
 training_mode = 1
-y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
+y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
 
 node = onnx.helper.make_node(
     'BatchNormalization',
     inputs=['x', 's', 'bias', 'mean', 'var'],
-    outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+    outputs=['y', 'output_mean', 'output_var'],
     training_mode=training_mode
 )
 
 # output size: (1, 2, 1, 3)
 expect(node, inputs=[x, s, bias, mean, var],
-       outputs=[y, output_mean, output_var, saved_mean, saved_var],
+       outputs=[y, output_mean, output_var],
        name='test_batchnorm_example_training_mode')
 
 # input size: (2, 3, 4, 5)
@@ -1399,20 +1399,20 @@ var = np.random.rand(3).astype(np.float32)
 training_mode = 1
 momentum = 0.9
 epsilon = 1e-2
-y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
-                                                                            epsilon)
+y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
+                                                      epsilon)
 
 node = onnx.helper.make_node(
     'BatchNormalization',
     inputs=['x', 's', 'bias', 'mean', 'var'],
-    outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+    outputs=['y', 'output_mean', 'output_var'],
     epsilon=epsilon,
     training_mode=training_mode
 )
 
 # output size: (2, 3, 4, 5)
 expect(node, inputs=[x, s, bias, mean, var],
-       outputs=[y, output_mean, output_var, saved_mean, saved_var],
+       outputs=[y, output_mean, output_var],
        name='test_batchnorm_epsilon_training_mode')
 ```
 

onnx/backend/test/case/node/batchnorm.py

@@ -29,7 +29,7 @@ def _batchnorm_training_mode(x, s, bias, mean, var, momentum=0.9, epsilon=1e-5):
     output_mean = mean * momentum + saved_mean * (1 - momentum)
     output_var = var * momentum + saved_var * (1 - momentum)
     y = _batchnorm_test_mode(x, s, bias, saved_mean, saved_var, epsilon=epsilon)
-    return y.astype(np.float32), saved_mean, saved_var, output_mean, output_var
+    return y.astype(np.float32), output_mean, output_var
 
 
 class BatchNormalization(Base):
@@ -84,18 +84,18 @@ def export_train():  # type: () -> None
         # using np.bool(1) while generating test data with "'bool' object has no attribute 'dtype'"
         # working around by using np.byte(1).astype(bool)
         training_mode = 1
-        y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
+        y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var)
 
         node = onnx.helper.make_node(
             'BatchNormalization',
             inputs=['x', 's', 'bias', 'mean', 'var'],
-            outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+            outputs=['y', 'output_mean', 'output_var'],
             training_mode=training_mode
         )
 
         # output size: (1, 2, 1, 3)
         expect(node, inputs=[x, s, bias, mean, var],
-               outputs=[y, output_mean, output_var, saved_mean, saved_var],
+               outputs=[y, output_mean, output_var],
                name='test_batchnorm_example_training_mode')
 
         # input size: (2, 3, 4, 5)
@@ -107,18 +107,18 @@ def export_train():  # type: () -> None
         training_mode = 1
         momentum = 0.9
         epsilon = 1e-2
-        y, saved_mean, saved_var, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
-                                                                                    epsilon)
+        y, output_mean, output_var = _batchnorm_training_mode(x, s, bias, mean, var, momentum,
+                                                              epsilon)
 
         node = onnx.helper.make_node(
             'BatchNormalization',
             inputs=['x', 's', 'bias', 'mean', 'var'],
-            outputs=['y', 'output_mean', 'output_var', 'saved_mean', 'saved_var'],
+            outputs=['y', 'output_mean', 'output_var'],
             epsilon=epsilon,
             training_mode=training_mode
         )
 
         # output size: (2, 3, 4, 5)
         expect(node, inputs=[x, s, bias, mean, var],
-               outputs=[y, output_mean, output_var, saved_mean, saved_var],
+               outputs=[y, output_mean, output_var],
                name='test_batchnorm_epsilon_training_mode')

onnx/backend/test/data/node/test_batchnorm_epsilon_training_mode/model.onnx

@@ -1,12 +1,11 @@
-backend-test:�
-�

+backend-test:�
+p
 
x
 
s
 bias
 mean
 var
youtput_mean
-output_var
-saved_mean	saved_var"BatchNormalization*
+output_var"BatchNormalization*
 epsilon
 �#<�

*
 training_mode
�
$test_batchnorm_epsilon_training_modeZ
@@ -45,14 +44,5 @@ saved_mean	saved_var"BatchNormalization*
 
 output_var
 
-

-b
-
-saved_mean
-
-

-b
-	saved_var
-
 

 B

onnx/backend/test/data/node/test_batchnorm_example_training_mode/model.onnx

@@ -1,12 +1,11 @@
-backend-test:�
-t
+backend-test:�
+]
 
x
 
s
 bias
 mean
 var
youtput_mean
-output_var
-saved_mean	saved_var"BatchNormalization*
+output_var"BatchNormalization*
 training_mode
�
$test_batchnorm_example_training_modeZ
 
x
 

@@ -43,14 +42,5 @@ saved_mean	saved_var"BatchNormalization*
 
 output_var
 
-

-b
-
-saved_mean
-
-

-b
-	saved_var
-
 

 B

onnx/checker.cc

@@ -619,7 +619,7 @@ void check_graph(const GraphProto& graph, const CheckerContext& ctx, const Lexic
             "Nodes in a graph must be topologically sorted, however input '",
             input,
             "' of node: \n",
-            ProtoDebugString(node),
+            "name: ", node.name(), " OpType: ", node.op_type(),
             "\n is not output of any previous nodes.");
       }
     }
@@ -633,7 +633,8 @@ void check_graph(const GraphProto& graph, const CheckerContext& ctx, const Lexic
     }
     ONNX_CATCH(ValidationError & ex) {
       ONNX_HANDLE_EXCEPTION([&]() {
-        ex.AppendContext("Bad node spec: " + ProtoDebugString(node));
+        ex.AppendContext(
+            "Bad node spec for node. Name: " + node.name() + " OpType: " + node.op_type());
         ONNX_THROW_EX(ex);
       });
     }
@@ -732,7 +733,7 @@ void check_function(const FunctionProto& function, const CheckerContext& ctx, co
             "Nodes in a function must be topologically sorted, however input '",
             input,
             "' of node: \n",
-            ProtoDebugString(node),
+            "Name: ", node.name(), " OpType: ", node.op_type(),
             "\n is neither output of any previous nodes nor input of the function.");
       }
     }

onnx/common/version.h

@@ -8,6 +8,6 @@
 namespace ONNX_NAMESPACE {
 
 // Represents the most recent release version. Updated with every release.
-constexpr const char* LAST_RELEASE_VERSION = "1.8.201";
+constexpr const char* LAST_RELEASE_VERSION = "1.8.202";
 
 }