Need attribute axis for rowwise_add, fc operator, etc.

[reyoung]

问题

我们开始预设一些Operator操作的输入Tensor维度是固定的。

譬如，

• rowwise_add op的输入是一个矩阵和一个向量。
• mul op的输入是两个矩阵

但在图像中，经常需要输入一个四维的Tensor，这四个维度是[N,C,H,W]。对于四维矩阵，也应该可以使用rowwise_add op加上bias。这时候我们需要将四维度的输入转换成二维度的输出。

实现方案对比

• 这种场景下我们需要给Op增加一个axis属性。例如，输入数据是四维[N,C,H,W]，将Axis设置成2，那么就会转换成二维的矩阵 [NC, HW]。即 new_shape = [product(shape[0:axis]), product(shape[axis:])]。

• 还有一种实现方式是，在rowwise_add op前后增加两个reshape操作，先把[N, C, H, W]的shape变换成[N*C, H*W]。再在rowwise_add op之后变换回来。这样做Op数量有点多，并不高效。

具体实现方法

• 在Eigen转换的过程中，给Matrix的转换增加一个int axis参数。默认是1。Operator可以自己设置axis。

[wangkuiyi]

Currently, our FCOp requires that its input tensor is 2-dimensional -- minibatch size times the number of features. It is often that the feature vector is a flattened image, which was 3-dimensional (channel, height, and width) originally. We need some way to flatten and input and to restore the output to the original shape.

We can do this by adding a pair of operators, FlattenOp and RestoreOp. A simpler alternative is to add an attribute helping the FCOp to flatten and restore.

For example, a mini-batch of images is represented by a 4-dimneional tensor of {N, C, H, W}. Usually, we have one weight matrix W which applies to each channel. In this case, we need to flatten the last two dimensions, H and W. Suppose that we want W applies to each image, we would need to platten the last three dimensional, C, H, and W. But anyway, we would platten the last n dimensions. We refer to n by a term NumFlattenDims.

Basically, we need to add an attribute num_flatten_dims to the FCOp, and similar operators including RowwiseAddOp.

[gongweibao]

参考一下caffe2的 element-wise binary multiplication的文档:
https://caffe2.ai/docs/operators-catalogue.html#mul

还有TensorFlow的：
https://www.tensorflow.org/api_docs/python/tf/multiply

我们也需要支持上述类似的方案吗？