/caffe

ReductionLayer #2089

Merged
merged 1 commit into from Jun 3, 2015
+599 −1

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
needs rebase ready for review
Projects
None yet
Milestone
No milestone
2 participants
@jeffdonahue @shelhamer
@jeffdonahue
Contributor

jeffdonahue commented Mar 10, 2015

Performs a "reduction" operation (currently SUM, MEAN, ASUM for sum of absolute values, and SUMSQ for sum of squares) to turn a number of "tail" axes into a single scalar value. The MEAN operation, in combination with a loss_weight, is useful for creating custom losses that don't have an obnoxious amount of output. For example, this EuclideanLoss:

layer {
  type: "EuclideanLoss"
  bottom: "preds"
  bottom: "targets"
  top: "mean_squared_error"
  loss_weight: 1
}

...is equivalent to this Reduction:

layer {
  type: "Eltwise"
  bottom: "preds"
  bottom: "targets"
  top: "error"
  eltwise_param {
    operation: SUM
    coeff: 1
    coeff: -1
  }
}
layer {
  type: "Reduction"
  bottom: "error"
  top: "sum_squared_error_per_instance"
  reduction_param { operation: SUMSQ axis: 1 }
}
layer {
  type: "Reduction"
  bottom: "sum_squared_error_per_instance"
  top: "mean_squared_error"
  reduction_param { operation: MEAN axis: 0 coeff: 0.5 }
  loss_weight: 1
}

(would be more efficient to do as a single Reduction with SUM_OF_SQUARES and a certain coeff setting, but then you have to compute the batch size and everything is less pretty...)

Eventually, this should support reduction along inner axes (e.g. support an end_axis), but that makes the implementation substantially more involved than this...

jeffdonahue added the ready for review label Mar 10, 2015

jeffdonahue changed the title from Reduction Layer to ReductionLayer Mar 10, 2015

@myfavouritekk myfavouritekk added a commit to myfavouritekk/caffe that referenced this pull request Mar 16, 2015

@myfavouritekk myfavouritekk Merge pull request #2089 from jeffdonahue/reduction-layer 
ReductionLayer

* jeffdonahue/reduction-layer:
  Add ReductionLayer to reduce any number of "tail" axes to a scalar value

Conflicts:
	src/caffe/proto/caffe.proto
 36a70ba

shelhamer added the needs rebase label Jun 1, 2015

@shelhamer shelhamer and 1 other commented on an outdated diff Jun 3, 2015

View full outdated diff
src/caffe/layers/reduction_layer.cpp
+ const Dtype* bottom_data = bottom[0]->cpu_data();
+ const Dtype* mult_data = NULL;
+ if (sum_multiplier_.count() > 0) {
+ mult_data = sum_multiplier_.cpu_data();
+ }
+ Dtype* top_data = top[0]->mutable_cpu_data();
+ for (int i = 0; i < num_; ++i) {
+ switch (op_) {
+ case ReductionParameter_ReductionOp_SUM:
+ case ReductionParameter_ReductionOp_MEAN:
+ *top_data = caffe_cpu_dot(dim_, mult_data, bottom_data);
+ break;
+ case ReductionParameter_ReductionOp_ASUM:
+ *top_data = caffe_cpu_asum(dim_, bottom_data);
+ break;
+ case ReductionParameter_ReductionOp_SUM_OF_SQUARES:
@shelhamer

shelhamer Jun 3, 2015

Owner

SUM_OF_SQUARES is a little unlike the other op names. SUMSQ fits in with ASUM. Your pick.

@jeffdonahue

jeffdonahue Jun 3, 2015

Contributor

fair enough -- I'll change it to SUMSQ

@shelhamer shelhamer and 1 other commented on an outdated diff Jun 3, 2015

View full outdated diff
src/caffe/test/test_reduction_layer.cpp
+ UniformFiller<Dtype> filler(filler_param);
+ filler.Fill(this->blob_bottom_);
+ blob_bottom_vec_.push_back(blob_bottom_);
+ blob_top_vec_.push_back(blob_top_);
+ }
+ virtual ~ReductionLayerTest() {
+ delete blob_bottom_;
+ delete blob_top_;
+ }
+
+ void TestForward(ReductionParameter_ReductionOp op, float coeff = 1) {
+ LayerParameter layer_param;
+ ReductionParameter* reduction_param = layer_param.mutable_reduction_param();
+ reduction_param->set_operation(op);
+ if (coeff != 1.0) {
+ reduction_param->set_coeff(2.3);
@shelhamer

shelhamer Jun 3, 2015

Owner

reduction_param->set_coeff(coeff); might be more robust for cases when coeff != 2.3.

@jeffdonahue

jeffdonahue Jun 3, 2015

Contributor

er, yeah, thanks, not sure what I was thinking when I wrote this...
@shelhamer
Owner

shelhamer commented Jun 3, 2015

Once the comments are addressed this looks fine, although there could be a test for reducing over a tail that isn't all dimensions.

p.s. N-D blobs are nice. count() and CanonicalAxisIndex()make for more obvious code.
@jeffdonahue
Contributor

jeffdonahue commented Jun 3, 2015

Thanks for all the reviews @shelhamer! Tests for non-zero axis added; will merge after Travis.
@jeffdonahue jeffdonahue Add ReductionLayer to reduce any number of "tail" axes to a scalar value 
Currently implements operations SUM, MEAN, ASUM (sum of absolute
values), and SUMSQ (sum of squares)
823d055
@jeffdonahue
Contributor

jeffdonahue commented Jun 3, 2015

Wow... especially thanks for the suggestion to test other axes. I had a bug in Backward for asum and sumsq (wasn't incrementing bottom_data). Should be good now...
@shelhamer
Owner

shelhamer commented Jun 3, 2015

Tests are more trustworthy than me. Good catch!

@jeffdonahue jeffdonahue added a commit that referenced this pull request Jun 3, 2015

@jeffdonahue jeffdonahue Merge pull request #2089 from jeffdonahue/reduction-layer 
ReductionLayer
 0cc7e18

@jeffdonahue jeffdonahue merged commit 0cc7e18 into BVLC:master Jun 3, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

jeffdonahue deleted the jeffdonahue:reduction-layer branch Jun 5, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment