/caffe

ND Crop layer #3570

Merged
merged 4 commits into from Mar 5, 2016

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement focus
Projects
None yet
Milestone
No milestone
5 participants
@BlGene @longjon @bjosv79 @shelhamer @elezar
@BlGene
Contributor

BlGene commented Jan 19, 2016

Something along the lines of what a ND crop layer would look like.
Based on: PR #1976 -> PR #3565 -> this

Todo

  • Test for D > 4
  • Test backwards
  • GPU kernel copy instead of memcopy (?)
  • const indices and offsets
  • GPU backwards/forwards
  • CPU backwards
  • parameter size checks
  • lint

Please see PR #3613 for how to set the crop parameters.
@BlGene
Contributor

BlGene commented Jan 20, 2016

@longjon So, my attempt at a ND crop layer, if this is approximately what you are looking for I can try to fix the GPU part.

Edit: For the GPU part just implementing the 4D case would seem like a reasonable compromise.
@longjon
Contributor

longjon commented Jan 21, 2016

Actually I'm confused about the parameter interface here... what I had in mind is something mirroring the conv layer interface, with a scalar (i.e., optional) param axis alongside a repeated param of crop values (as you have currently).

The way it would work would be the same way conv layer works: if only one crop value is specified, all the dimensions after the specified axis would be cropped (by the same amount); otherwise the number of crop values specified must be equal to the number of dimensions following axis, and the trailing dimensions would be cropped accordingly.

The reason for doing it this way is because the crop values need to be determined from the net specification, and the net specification doesn't know how many dimensions the data will have! If the layers we are "cropping through" include conv/pool layers with non-singleton kernel sizes, then we know exactly how many dimensions to crop; but if not, all we can do is propagate the axis param and determine how many dimensions to crop at run time. (At least, I think that should work.)

It would be nice to have an ND GPU kernel, but it's fine to punt that to get the basic functionality in.

Thanks!
@BlGene
Contributor

BlGene commented Jan 21, 2016

@longjon The parameters should do what you want now.

I substituted the copy kernel with repeated calls to caffe_copy, which if I understood correctly will call cudaMemcpy, and thus perform the copy operation on the GPU. My intuition is that calling this thousands of times is probably less efficient than writing a custom kernel.
@longjon
Contributor

longjon commented Jan 22, 2016

The parameters look right now.

caffe_copy will perform the copy on the GPU, but (as you suggest) has overhead -- even in the 2D case I found the overhead unacceptable, which is why the kernel exists. A fine solution for now would be to keep the existing kernel around, and use it only for the innermost two dimensions, looping around the outer ones. That should keep the performance the same in the 2D case while supporting ND.
@BlGene
Contributor

BlGene commented Jan 26, 2016

@longjon The crop_copy_gpu function now calls the kernel for the innermost two dimensions. This should hopefully fix the speed problem.

longjon referenced this pull request Jan 30, 2016

Merged

Python/net spec coordinate map and crop computation #3613

8 of 8 tasks complete

BlGene changed the title from ND Crop layer [WIP] to ND Crop layer Feb 1, 2016

@bjosv79

bjosv79 commented Feb 19, 2016

It seems to a few problems with the backward pass of crop_layer.cu:

  • A typo src_outer_stride instead of src_inner_stride in line 94.
  • Incorrect number of elements in caffe_gpu_set in line 89.
  • Is the caffe_gpu_set really necessary? It is already done in line 119?
@BlGene
Contributor

BlGene commented Feb 19, 2016

@bjosv79 Thanks. All those things seemed correct, I changed them ( but haven't tested yet ).

@shelhamer shelhamer added enhancement JL ES labels Feb 25, 2016

@shelhamer shelhamer commented on an outdated diff Feb 27, 2016

View full outdated diff
src/caffe/proto/caffe.proto
@@ -306,7 +306,7 @@ message ParamSpec {
// NOTE
// Update the next available ID when you add a new LayerParameter field.
//
-// LayerParameter next available layer-specific ID: 143 (last added: scale_param)
+// LayerParameter next available layer-specific ID: 144 (last added: crop_param)
@shelhamer

shelhamer Feb 27, 2016

Owner

Needs ID++ after #3211 -- sorry for the rebase.

@shelhamer shelhamer commented on an outdated diff Feb 27, 2016

View full outdated diff
include/caffe/layers/crop_layer.hpp
@@ -0,0 +1,67 @@
+#ifndef CAFFE_CROP_LAYER_HPP_
+#define CAFFE_CROP_LAYER_HPP_
+
+#include <utility>
+#include <vector>
+
+#include "caffe/blob.hpp"
+#include "caffe/layer.hpp"
+#include "caffe/proto/caffe.pb.h"
+
+namespace caffe {
+
+/**
+ * @brief Takes a Blob and crop it along either the width or height dimension,
@shelhamer

shelhamer Feb 27, 2016

Owner

This doc comment is out-of-date now that the layer is N-D and cropping is not one dimension or another, but across all dimensions after the axis.

@shelhamer shelhamer commented on an outdated diff Feb 27, 2016

View full outdated diff
src/caffe/layers/crop_layer.cpp
+
+template <typename Dtype>
+void CropLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom,
+ const vector<Blob<Dtype>*>& top) {
+ CHECK_EQ(bottom.size(), 2) << "Wrong number of bottom blobs.";
+ // parameter setup moved to Reshape because it depends on size.
+}
+
+template <typename Dtype>
+void CropLayer<Dtype>::Reshape(const vector<Blob<Dtype>*>& bottom,
+ const vector<Blob<Dtype>*>& top) {
+ const CropParameter& param = this->layer_param_.crop_param();
+ // bottom[0] supplies the data
+ // bottom[1] supplies the size
+ int input_dim = bottom[0]->num_axes();
+ CHECK_LT(param.axis(), input_dim) << "crop axis bigger than input dim";
@shelhamer

shelhamer Feb 27, 2016

Owner

It seems safe to move any logic that depends only on the number of dimensions to LayerSetUp(); switching the number of dims. would break many layers.

@shelhamer shelhamer commented on an outdated diff Feb 27, 2016

View full outdated diff
src/caffe/layers/crop_layer.cpp
+ }
+ top[0]->Reshape(new_shape);
+}
+
+// recursive copy function
+template <typename Dtype>
+void CropLayer<Dtype>::crop_copy(const vector<Blob<Dtype>*>& bottom,
+ const vector<Blob<Dtype>*>& top,
+ const vector<int>& offsets,
+ vector<int> indices,
+ int cur_dim,
+ const Dtype* src_data,
+ Dtype* dest_data,
+ bool is_forward) {
+ if (cur_dim + 1 < top[0]->num_axes()) {
+ // We are not yet at the final dimension, call copy recursivley
@shelhamer

shelhamer Feb 27, 2016

Owner

recursivley -> recursively

@shelhamer shelhamer commented on an outdated diff Feb 27, 2016

View full outdated diff
src/caffe/layers/crop_layer.cu
+ bottom[0]->offset(ind_off);
+ // NOLINT_NEXT_LINE(whitespace/operators)
+ copy_kernel<<<CAFFE_GET_BLOCKS(lines), CAFFE_CUDA_NUM_THREADS>>>(
+ lines, height, width,
+ dest_outer_stride, dest_inner_stride,
+ src_outer_stride, src_inner_stride,
+ top_diff, bottom_diff);
+ }
+ }
+}
+
+template <typename Dtype>
+void CropLayer<Dtype>::Forward_gpu(const vector<Blob<Dtype>*>& bottom,
+ const vector<Blob<Dtype>*>& top) {
+ std::vector<int> indices(top[0]->num_axes(), 0);
+ // This works because crop_copy uses caffe_copy which calls cudaMemcpy.
@shelhamer

shelhamer Feb 27, 2016

Owner

Drop out-of-date comment as the implementation now uses a copy kernel.

shelhamer added the focus label Feb 27, 2016

This was referenced Feb 27, 2016

Closed
Closed
@shelhamer
Owner

shelhamer commented Feb 27, 2016

@BlGene I pushed a rebase on the latest master to my fork https://github.com/shelhamer/caffe/tree/crop-nd with a few other additions -- feel free to integrate my changes if the commits save you time.

@shelhamer shelhamer commented on an outdated diff Feb 28, 2016

View full outdated diff
src/caffe/proto/caffe.proto
@@ -597,6 +598,16 @@ message ConvolutionParameter {
optional bool force_nd_im2col = 17 [default = false];
}
+message CropParameter {
+ // If only one crop_offset is specified, all the dimensions after the specified
+ // axis would be cropped (by the same amount); otherwise the number of crop
+ // values specified must be equal to the number of dimensions following axis, and
+ // the trailing dimensions would be cropped accordingly.
+ // Protip: standard dimensions are ( N,C,H,W )
+ optional uint32 axis = 1 [default = 2];
@shelhamer

shelhamer Feb 28, 2016

Owner

Let's make axis mirror the axis arg of other layers to allow for negative indexing through an int32 field and the CanonicalAxisIndex() helper method? See for instance https://github.com/BVLC/caffe/blob/master/src/caffe/layers/flatten_layer.cpp#L12-L13

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Feb 28, 2016

@shelhamer shelhamer crop -> offset to fit #3570 81a6a6a
@BlGene
Contributor

BlGene commented Feb 29, 2016

@shelhamer this last commit should address all the points you mentioned.

This was referenced Feb 29, 2016

Open
Merged

@shelhamer shelhamer commented on an outdated diff Mar 1, 2016

View full outdated diff
src/caffe/layers/crop_layer.cpp
+void CropLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom,
+ const vector<Blob<Dtype>*>& top) {
+ // All logic that depends only on the number of dimensions is here,
+ // the rest is in Reshape because it depends on Blob size.
+ // bottom[0] supplies the data
+ // bottom[1] supplies the size
+ const CropParameter& param = this->layer_param_.crop_param();
+ CHECK_EQ(bottom.size(), 2) << "Wrong number of bottom blobs.";
+ int input_dim = bottom[0]->num_axes();
+ const int start_axis = bottom[0]->CanonicalAxisIndex(param.axis());
+ CHECK_LT(start_axis, input_dim) << "crop axis bigger than input dim";
+ if (param.offset_size() > 1) {
+ // the number of crop values specified must be equal to the number
+ // of dimensions following axis
+ CHECK_EQ(start_axis + param.offset_size(), input_dim)
+ << "number of crop values specified must be equal to the number of "
@shelhamer

shelhamer Mar 1, 2016

Owner

crop values -> offset values
@shelhamer
Owner

shelhamer commented Mar 1, 2016

Other then the last trivial comment all that's needed now are tests. @BlGene do you have these planned, or should I take a stab at it?
@BlGene
Contributor

BlGene commented Mar 1, 2016

@shelhamer
Fixed crop values -> offset values
Added tests.

BlGene referenced this pull request Mar 1, 2016

Closed

Simplified CropLayer #3565

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 2, 2016

@shelhamer shelhamer crop -> offset to fit #3570 15dc5d8

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 2, 2016

@shelhamer shelhamer adjust crop axis by 1 to fit #3570 d155544

@shelhamer shelhamer added a commit that referenced this pull request Mar 3, 2016

@shelhamer shelhamer crop -> offset to fit #3570 464939f

@shelhamer shelhamer added a commit that referenced this pull request Mar 3, 2016

@shelhamer shelhamer adjust crop axis by 1 to fit #3570 d2b85df

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 3, 2016

@shelhamer shelhamer crop -> offset to fit #3570 1f8f213

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 3, 2016

@shelhamer shelhamer adjust crop axis by 1 to fit #3570 d5a051a

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer crop -> offset to fit #3570 f4c762e

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer adjust crop axis by 1 to fit #3570 fee422d

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 988b98c
@shelhamer
Owner

shelhamer commented Mar 4, 2016

@BlGene I revised the tests in my latest commits to shelhamer/caffe:nd-crop to

  • align with test conventions
  • add a 5D test
  • switch to the standard gradient checker

so that all the TODOs are covered. Can you incorporate my changes into this branch for merge? Please squash commits to simplify history (including mine, I don't mind) into something like

  • a40439d original crop layer (longjon)
  • 1e9d279 extend to N-D (BlGene) with 409fa12, 9a4c15c squashed
  • 6437271 tests + canonical axis (BlGene)
  • test grooming: style, 5D, standard grad check (shelhamer)

I've added tests to #3613 so we can merge both once this is ready.

@shelhamer shelhamer removed ES JL labels Mar 4, 2016

@BlGene
Contributor

BlGene commented Mar 4, 2016

@shelhamer Squashed!
@shelhamer
Owner

shelhamer commented Mar 4, 2016

@BlGene thanks for the quick reply! Sorry to be so obsessed with history but can you reword the messages to make the subject line (first line of the message) more complete? That is, b6ee629 could be "Crop: fixes, tests, and negative axis indexing" and 631ff09 could be "Crop: more tests and test tuning" with the rest of the details in a list.

With that this will be a model PR and finally merged! Thanks for all your work on this and the attention to detail. I'm excited to have all the FCN tools in master soon.

shelhamer closed this Mar 4, 2016

shelhamer reopened this Mar 4, 2016

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 6fa55a1

@shelhamer shelhamer added a commit to shelhamer/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer Merge pull request #3570 from BlGene/crop-nd 
ND Crop layer

* BlGene/crop-nd:
  Crop: reduce test blob dims for speed Crop test: standard Gaussian filler, only blob_bottom_vec_            polish formatting, rename tests, test HW crop, test 5D crop            standard gradient checks
  Added fixes suggested by Evan, including negative indexing. Added layer tests.
  Extend Crop to N-D, changed CropParameter.
  add CropLayer: crop blob to another blob's dimensions with offsets
 05a3216

@shelhamer shelhamer added a commit to shelhamer/caffe that referenced this pull request Mar 4, 2016

@shelhamer shelhamer Merge pull request #3613 from longjon/py-coord-map 
Python/net spec coordinate map and crop computation

* longjon/py-coord-map:
  [pycaffe] test coord_map
  [pycaffe] align coord_map and #3570 Crop layer
  [pycaffe] document, style, and complete coord_map
  [pycaffe] add coord_map.py for computing induced coordinate transform
 7135e3c

@shelhamer shelhamer added a commit to longjon/caffe that referenced this pull request Mar 5, 2016

@shelhamer shelhamer [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 25b9ef9
@BlGene
Contributor

BlGene commented Mar 5, 2016

@shelhamer OK, I fixed the commit messages.

I'm happy to have FCN functionality merged too. Thanks for the attention to detail. 👍

longjon and others added some commits Dec 27, 2014

@longjon @BlGene longjon add CropLayer: crop blob to another blob's dimensions with offsets 
configure offset(s) through proto definition.
 64e78bd
@BlGene BlGene Extend Crop to N-D, changed CropParameter. 952fd17
@BlGene BlGene Crop: fixes, tests and negative axis indexing. ca9fa49
@shelhamer @BlGene shelhamer Crop: more tests and test tuning. 
Changes are:
  reduce test blob dims for speed
  use standard Gaussian filler,
  polish formatting and rename tests,
  test HW crop and 5D crop,
  standard gradient checks
e03a287

@shelhamer shelhamer added a commit that referenced this pull request Mar 5, 2016

@shelhamer shelhamer Merge pull request #3570 from BlGene/crop-nd 
ND Crop layer
 ca1ce4a

@shelhamer shelhamer merged commit ca1ce4a into BVLC:master Mar 5, 2016
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

BlGene deleted the BlGene:crop-nd branch Mar 6, 2016

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Mar 17, 2016

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 bdb72aa
@elezar
Contributor

elezar commented Mar 31, 2016

Just a quick question from my side, can the ND crop be used to implement a rectangular crop as in #1980?

elezar referenced this pull request Mar 31, 2016

Closed

Non-square cropping #1980

@BlGene
Contributor

BlGene commented Mar 31, 2016

This layer crops blob B to have the same shape as Blob A. So its a bit hacky, but you could create a dummy blob (A) that has the desired shape and crop the (B) blob to this size.

BR, Max

@SvenTwo SvenTwo added a commit to SvenTwo/caffe that referenced this pull request Apr 6, 2016

@shelhamer @SvenTwo shelhamer + SvenTwo [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 84b3f07

@emaggiori emaggiori pushed a commit to emaggiori/caffe that referenced this pull request Apr 8, 2016

@shelhamer shelhamer + emmanuel maggiori [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 6fa5155

This was referenced Apr 10, 2016

Closed
Merged

@sato-cloudian sato-cloudian added a commit to sato-cloudian/caffe that referenced this pull request Aug 10, 2016

@shelhamer @sato-cloudian shelhamer + sato-cloudian [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 7dbb645

@fxbit fxbit added a commit to Yodigram/caffe that referenced this pull request Sep 1, 2016

@shelhamer @fxbit shelhamer + fxbit [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 d49d210

@fxbit fxbit added a commit to Yodigram/caffe that referenced this pull request Sep 1, 2016

@shelhamer @fxbit shelhamer + fxbit Merge pull request #3570 from BlGene/crop-nd 
ND Crop layer
 4cdf891

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Oct 24, 2016

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 9ee8fb3

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Oct 24, 2016

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 d9ab5a7

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Feb 15, 2017

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 f0effaf

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Feb 15, 2017

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 fa42eb2

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Feb 15, 2017

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 518f9aa

@zouxiaochuan zouxiaochuan added a commit to zouxiaochuan/caffe that referenced this pull request Feb 15, 2017

@shelhamer @zouxiaochuan shelhamer + zouxiaochuan [pycaffe] align coord_map and #3570 Crop layer 
- crop -> offset
- adjust crop axis by 1
 0c6d453
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment