Blobs are N-D arrays (for N not necessarily equals 4) #1970

jeffdonahue · Feb 25, 2015

Replaces #1486 -- this PR is to master instead of dev. This is rebased and is ready for review @longjon.

This PR gives Blobs a vector<int> shape_ of dimensions, rather than the old num, channels, height, width. The first commit is all the changes needed to get Caffe to compile and everything to run as before with the new vector of tensor dimensions. The remaining commits generalize some existing classes to use the new tensor dimensions (but they are not necessary to make it run, as it's still fine to just use all 4-D blobs with extra singleton dimensions where needed).

jeffdonahue · Mar 2, 2015

I had added a commit intended to add matcaffe support for N-D array blobs, but decided to defer to #1913 to add the support, as I don't have much experience with matcaffe and it might require some special cases (e.g., I remembered that MATLAB doesn't support arrays with <2 axes so would need to figure out what to do with those cases -- maybe the MATLAB default of adding extra axes with dimension 1 would work fine, but not sure). But if that commit might be helpful in the development of #1913, feel free to cherry-pick or just refer to it here (but note that I didn't test or even try to compile it...).

So for now, matcaffe will continue to work fine for blobs with <= 4 axes, but will die on attempts to access blobs with >4 dimensions (per their use of num/channels/height/width, which call LegacyShape).

My 2 cents - uint32 should be enough? A 2G blob (8G in floats) looks big enough.

(But yeah, 640k used to be enough for everything.)

My feeling is that it's probably worth future-proofing given the relatively tiny amount of extra storage cost (unless you're really using blobs with INT_MAX axes, as are now allowed...). I could make it uint32 for now and then later change it to int64 if/when we really find practical uses for blobs that large, but that might cause backward compatibility problems with (de)serialization that I don't even want to think about... (Why not uint64? I think it's worth keeping -1 and other <0 values reserved for various purposes -- e.g. setting axis [default = -1] to signify "all axes", or that an axis must be manually specified and should fail otherwise, etc. And it does seem pretty unlikely that in the reasonably foreseeable future we'll need 2^64 byte blobs, so 2^63-1 bytes seems like an okay max...)

I'm open to discussion on any of this though.

Future-proof sounds better :) Thanks for checking with me!

But Jeff, there is no reason for any individual to have a > 2GB blob in their home. Haha no, I'm all for int64 dimensions.

longjon · Mar 2, 2015

Oops, just realized that I've been commenting on commits again. Agree with int64 blob dimensions. I'm ready to merge this today, pending comments made, the new type for Python Blob.reshape (which I'll do), and a nod from @shelhamer.

longjon · Mar 2, 2015

Added my Python update... this should be ready pending your approval of that, Travis, and @shelhamer.

jeffdonahue · Mar 3, 2015

Thanks Jon! Your updated Blob_Reshape looks good to me.

shelhamer · Mar 3, 2015

The axis field in the proto is inconsistently (u)int32 -- should it be int32 everywhere to allow positive and negative indexing in Slice, Concat, and InnerProduct dimension picking? Or it's fine to keep this the same for now as current behavior.

For example, if bottom[0]'s shape is (N, C, H, W) and axis = 1

shelhamer · Mar 3, 2015

@jeffdonahue this is sweet. My only comments were minor, so do what you want with them and merge!

This could be a good time to add a check that the prediction and target dimensions (except channel) agree?

done in a717963 -- I was a little less strict about it, just checking that the number of labels matches the number of predictions, as I couldn't figure out a way to do it that's backward compatible and nice to use in the future. (For example, I didn't want to break the current behavior of (NxCxHxW) predictions with (Nx1xHxW) labels, but it seems silly to require the singleton axis in the labels, but requiring (NxHxW) labels -- no singleton axis -- breaks backwards compatibility.)

jeffdonahue · Mar 3, 2015

Hey Evan, thanks for the review! I made the additional commits above in response to your comments -- ~~currently marked with fixup! to be autosquashed later~~. I thought you might want to take a look since, re negative indexing, I applied it in SliceLayer, ConcatLayer, SoftmaxLayer, and InnerProductLayer, and ended up adding an additional Blob method CanonicalAxisIndex (as I started to realize I was writing the same checks several times). I added a single test that actually uses it to TestConcatLayer.

Edit: I went ahead and squashed the fixups, but I saved the state before the squash in another branch "tensor-blob-presquash" if you still want to look at only the additional changes I made after your review.

jeffdonahue · Mar 4, 2015

This should be ready to go -- I'll merge as soon as @shelhamer or @longjon quickly OKs the small fixups I made last night in response to @shelhamer's suggestions (can see just those changes at jeffdonahue/caffe@eca584d...jeffdonahue:tensor-blob-presquash).

bhack · Mar 4, 2015

This will probably cross impact every new layer in the PRs queue. How many new layers we have in the "waiting list"? The merging order is not neutral because will impact the adaptation charge to developer of one PR instead of another.

shelhamer · Mar 4, 2015

Thanks for the fixups @jeffdonahue -- this all looks good to me. This PR can be a guide to converting in-progress and future layers to N != 4. I'm happy to see brewing transcend to higher dimensions.

Next up: N-D convolution and pooling!

ddetone · Mar 4, 2015

@shelhamer Is there an existing branch in which someone has begun N-D convolution and pooling? I am currently working on my own 3-D convolution and pooling implementation.

BestSonny · Mar 7, 2015

MY data is 64 ..36..1..1 and label is 64..1..1..1
I guess the label must be changed to 64 ..36..1..1
Or I'll get the following errors.
Check failed: bottom[0]->shape(i) <= bottom[1]->shape(i) (36 vs. 1) Dimension mismatch between predictions and label

Can you give some tips?

Is there some thing wrong?
CHECK_LE(bottom[0]->shape(i), bottom[1]->shape(i))

top[0]->Reshape(1, 1, 1, 1); + << "Dimension mismatch between predictions and label.";?

shelhamer · Mar 7, 2015

@jeffdonahue can you align https://github.com/BVLC/caffe/blob/master/docs/tutorial/net_layer_blob.md with the transcendence to N-D blob sometime?

dbcam · Jan 14, 2016

This code implies that 'index' can be negative, however, line 150 indexes a std::vector<> that does not support negative indices. Blob::ShapeEquals() calls LegacyShape(index) with a negative index. Should probably drop pseudo-support for negative indices [Line 143 change -4 to 0. Line 144 delete '|| index < -num_axes()'.]

shape is a function. shape_ is the std::vector. So it is ok to have a negative index in line 150, since shape calls CanonicalAxisIndex.

You are correct Sean. My mistake. I confused array indexing with function call syntax. Thank you.

seanbell · Jan 16, 2016

I think this is ok since LegacyShape can handle negative arguments (see other reply to your other note).

shelhamer added JL ES ready for review labels Feb 25, 2015

This was referenced Feb 28, 2015

Closed

Xavier filler and inner product parameters #1575

Closed

MSRA weight filler #1946

shelhamer referenced this pull request Mar 2, 2015
Closed
Matcaffe2 #1913

shelhamer added a commit that referenced this pull request Mar 4, 2015

shelhamer Merge pull request #1970 from jeffdonahue/tensor-blob
Blobs are N-D arrays (for N not necessarily equals 4)
85bb397

shelhamer merged commit 85bb397 into BVLC:master Mar 4, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

jeffdonahue added a commit that referenced this pull request Mar 4, 2015

jeffdonahue fix comment I forgot about from @shelhamer's review of #1970 dec148e

jeffdonahue deleted the jeffdonahue:tensor-blob branch Mar 4, 2015

This was referenced Mar 5, 2015

Closed

Python classifier.py crash when running classification.ipynb #2041

Closed

Fixed issue 2041, updating code to match pull 1970 #2046

shelhamer added a commit to shelhamer/caffe that referenced this pull request Mar 7, 2015

shelhamer [pycaffe] no need to squeeze output after #1970
fix #2041 reported by @dgmp88
38ec847

This was referenced Mar 7, 2015

Closed

Fixed up accuracy layer. #1942

Closed

Updates on API documentation #1411

shelhamer referenced this pull request Mar 8, 2015
Closed
Preserve shape of extracted features #1457

shelhamer added a commit that referenced this pull request Mar 8, 2015

shelhamer [pycaffe] no need to squeeze output after #1970
fix #2041 reported by @dgmp88
fc35930

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 32e6e96

This was referenced Mar 9, 2015

Merged

Fixup AccuracyLayer like SoftmaxLossLayer in #1970 #2076

Closed

How to use the new accuracy #2063

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 a3e4604

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 3385d1c

jeffdonahue referenced this pull request Mar 9, 2015
Merged
FlattenLayer gets a FlattenParameter with an axis, end_axis #2082

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 6d5c8b2

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 7a40f74

jeffdonahue added a commit that referenced this pull request Mar 9, 2015

jeffdonahue Merge pull request #2076 from jeffdonahue/accuracy-layer-fixes
Fixup AccuracyLayer like SoftmaxLossLayer in #1970
77ab8f6

jeffdonahue referenced this pull request Mar 10, 2015
Closed
Very simple version of ReshapeLayer #2088

qinhongwei pushed a commit to qinhongwei/caffe that referenced this pull request Mar 12, 2015

cv [pycaffe] no need to squeeze output after #1970, add change to detect…
…or.py
844bdb6

seanbell referenced this pull request Mar 16, 2015
Closed
Extract Features from fc7 #700

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

jeffdonahue Update docs for ND blobs (#1970) and layer type is a string (#1694) f10c43b

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

jeffdonahue Update docs for ND blobs (#1970) and layer type is a string (#1694) 0afd1bf

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

jeffdonahue Update docs for ND blobs (#1970) and layer type is a string (#1694) a9d4adc

pclove1 referenced this pull request Apr 7, 2015
Closed
fix #1362. prefetch HDF5DataLayer #2271

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request May 15, 2015

jeffdonahue + shelhamer Update docs for ND blobs (#1970) and layer type is a string (#1694) cf0e6b7

shelhamer added a commit that referenced this pull request May 15, 2015

shelhamer Merge pull request #2201 from jeffdonahue/tutorial-fixes
Update docs for ND blobs (#1970) and layer type is a string (#1694)
af224c1

ddetone added a commit to ddetone/caffe that referenced this pull request Jun 26, 2015

jeffdonahue + ddetone Update docs for ND blobs (#1970) and layer type is a string (#1694) 7841337

matthiasplappert added a commit to matthiasplappert/caffe that referenced this pull request Aug 10, 2015

jeffdonahue + matthiasplappert Update docs for ND blobs (#1970) and layer type is a string (#1694) a7b2aab

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

shelhamer + cbfinn [pycaffe] no need to squeeze output after #1970
fix #2041 reported by @dgmp88
0bde346

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

jeffdonahue + cbfinn Fixup AccuracyLayer like SoftmaxLossLayer in #1970 -- fixes #2063 d37d928

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

jeffdonahue + cbfinn Update docs for ND blobs (#1970) and layer type is a string (#1694) 3bd95a4

This was referenced Aug 20, 2015

Merged

Use input_shape instead of input_dim in examples #2950

Merged

bugfix for ConcatLayer with propagate_down set #2972

Merged

Add argmax_param "axis" to maximise output along the specified axis #3069

wangyida added a commit to wangyida/caffe that referenced this pull request Sep 22, 2015

jeffdonahue + wangyida Update docs for ND blobs (#1970) and layer type is a string (#1694) 233c981

This was referenced Oct 7, 2015

Closed

Batch Normalization Layer #3161

Closed

Training with HDF5: blob size exceeds INT_MAX #3510

@@ -2,13 +2,21 @@ syntax = "proto2";
      package caffe;
+    +// Specifies the shape (dimensions) of a Blob.
+    +message BlobShape {
+    +  repeated int64 dim = 1 [packed = true];

@@ -15,7 +15,11 @@ void InnerProductLayer<Dtype>::LayerSetUp(const vector<Blob<Dtype>*>& bottom,
        const int num_output = this->layer_param_.inner_product_param().num_output();
        bias_term_ = this->layer_param_.inner_product_param().bias_term();
        N_ = num_output;
-    -  K_ = bottom[0]->count() / bottom[0]->num();
+    +  const int dim = this->layer_param_.inner_product_param().axis();
+    +  // Dimensions starting from "axis" are "flattened" into a single
+    +  // length K_ vector. For example, if bottom[0]'s shape is (N, C, H, W),

@@ -35,6 +35,9 @@ void SoftmaxWithLossLayer<Dtype>::Reshape(
          const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
        LossLayer<Dtype>::Reshape(bottom, top);
        softmax_layer_->Reshape(softmax_bottom_vec_, softmax_top_vec_);
+    +  softmax_axis_ = this->layer_param_.softmax_param().axis();
+    +  outer_num_ = bottom[0]->count(0, softmax_axis_);
+    +  inner_num_ = bottom[0]->count(softmax_axis_ + 1);

jeffdonahue	`Blobs are ND arrays (for N not necessarily equals 4).` vector<int> shape_ instead of (num, channels, height, width).	`1434e87`
jeffdonahue	`InnerProductLayer weights are 2D; biases are 1D`	`559ff3a`
jeffdonahue	`LossLayer output is 0D (scalar)`	`a0fa2a9`
jeffdonahue	`InnerProductLayer can multiply along any axis`	`29581e6`
jeffdonahue	`TestBlob: test that legacy BlobProtos are correctly handled by ShapeE…` …quals	`c4e9ec4`
jeffdonahue	`ConvLayer biases are 1D`	`94179cc`
jeffdonahue	`Add BlobShape message; use for Net input shapes`	`5407f82`
jeffdonahue	`add offset, {data,diff}_at nd blob accessors`	`119a1c6`
jeffdonahue	`Fix sparse GaussianFiller for new IPLayer weight axes`	`cf9fdda`
jeffdonahue	`EltwiseLayer need not assume old 4D dim names`	`bf73cb1`
jeffdonahue	`ConcatLayer: generalized Blob axes`	`8afdcd0`
jeffdonahue	`FlattenLayer: generalized Blob axes`	`1b97c06`
jeffdonahue	`TestConcatLayer: fix style errors`	`704e524`
jeffdonahue	`TestConcatLayer: add forward/gradient tests for concatenation along num`	`d52e9a8`
jeffdonahue	`SliceLayer: generalized Blob axes`	`b868916`
jeffdonahue	`common_layers.hpp: remove unused "Blob col_bob_"`	`fb9caee`
jeffdonahue	`AccuracyLayer output is 0D (scalar)`	`d8c6aeb`
jeffdonahue	`AccuracyLayer generalized to N instance axes`	`6b8a765`
jeffdonahue	`Test{Net,Solver} fixes for AccuracyLayer generalization`	`8e96445`
jeffdonahue	`SoftmaxLayer: generalized Blob axes`	`abec302`
jeffdonahue	`SplitLayer: change Reshape(n,h,c,w) to ReshapeLike(...)`	`e2bc9f9`
jeffdonahue	`DataLayer outputs 1D labels`	`e56377d`
jeffdonahue	`HDF5DataLayer shapes output according to HDF5 shape`	`e6468e9`
jeffdonahue	`MemoryDataLayer outputs 1D labels`	`7c8725b`
jeffdonahue	`ImageDataLayer outputs 1D labels`	`c87a136`
jeffdonahue	`WindowDataLayer outputs 1D labels`	`9505001`
jeffdonahue	`EuclideanLossLayer: generalized Blob axes`	`fcbb933`
jeffdonahue	`DummyDataLayer outputs blobs of arbitrary shape`	`7462c84`
jeffdonahue	`PyBlobs support generalized axes`	`269dafa`
jeffdonahue	`Add CHECK_EQ(4, ...)s to "vision layers" to enforce that the` num/channnels/height/width indexing is valid.	`69fc1f6`
jeffdonahue	`Add option not to reshape to Blob::FromProto; use when loading Blobs` from saved NetParameter Want to keep the param Blob shape the layer has set, and not necessarily adopt the one from the saved net (e.g. want to keep new 1D bias shape, rather than take the (1 x 1 x 1 x D) shape from a legacy net).	`71df6f9`
jeffdonahue	`SoftmaxLossLayer generalized like SoftmaxLayer`	`94d93da`
jeffdonahue	`CuDNNSoftmaxLayer: generalized Blob axes`	`60c288b`
longjon	`[pycaffe] expose Blob.reshape as *args function`	`aa242aa`
longjon	`[pytest] use non-4d blobs in test_python_layer`	`8c79d65`

BVLC/caffe

Join GitHub today

Blobs are N-D arrays (for N not necessarily equals 4) #1970

Conversation

jeffdonahue commented Feb 25, 2015

shelhamer added JL ES ready for review labels Feb 25, 2015

This was referenced Feb 28, 2015

jeffdonahue commented Mar 2, 2015

Yangqing commented on the diff Mar 2, 2015

shelhamer referenced this pull request Mar 2, 2015

longjon commented Mar 2, 2015

longjon commented Mar 2, 2015

jeffdonahue commented Mar 3, 2015

shelhamer commented Mar 3, 2015

Show 1 comment Hide comment shelhamer commented on an outdated diff Mar 3, 2015

shelhamer commented Mar 3, 2015

shelhamer commented on the diff Mar 3, 2015

jeffdonahue commented Mar 3, 2015

jeffdonahue and others added some commits Nov 26, 2014

jeffdonahue commented Mar 4, 2015

bhack commented Mar 4, 2015

shelhamer commented Mar 4, 2015

shelhamer added a commit that referenced this pull request Mar 4, 2015

Hide details View details shelhamer merged commit 85bb397 into BVLC:master Mar 4, 2015 1 check passed

1 check passed

ddetone commented Mar 4, 2015

jeffdonahue added a commit that referenced this pull request Mar 4, 2015

jeffdonahue deleted the jeffdonahue:tensor-blob branch Mar 4, 2015

This was referenced Mar 5, 2015

shelhamer added a commit to shelhamer/caffe that referenced this pull request Mar 7, 2015

This was referenced Mar 7, 2015

BestSonny commented on d8c6aeb Mar 7, 2015

BestSonny replied Mar 7, 2015

shelhamer commented Mar 7, 2015

shelhamer referenced this pull request Mar 8, 2015

shelhamer added a commit that referenced this pull request Mar 8, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

This was referenced Mar 9, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue referenced this pull request Mar 9, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

jeffdonahue added a commit that referenced this pull request Mar 9, 2015

jeffdonahue referenced this pull request Mar 10, 2015

qinhongwei pushed a commit to qinhongwei/caffe that referenced this pull request Mar 12, 2015

seanbell referenced this pull request Mar 16, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

pclove1 referenced this pull request Apr 7, 2015

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request May 15, 2015

shelhamer added a commit that referenced this pull request May 15, 2015

myfavouritekk added a commit to myfavouritekk/caffe that referenced this pull request May 15, 2015

This was referenced May 17, 2015

ddetone added a commit to ddetone/caffe that referenced this pull request Jun 26, 2015

matthiasplappert added a commit to matthiasplappert/caffe that referenced this pull request Aug 10, 2015

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

cbfinn added a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

This was referenced Aug 20, 2015

wangyida added a commit to wangyida/caffe that referenced this pull request Sep 22, 2015

This was referenced Oct 7, 2015

dbcam commented on include/caffe/blob.hpp in 1434e87 Jan 14, 2016

seanbell replied Jan 16, 2016

dbcam replied Jan 17, 2016

seanbell commented on src/caffe/blob.cpp in 1434e87 Jan 16, 2016

shelhamer commented on an outdated diff Mar 3, 2015

shelhamer merged commit `85bb397` into BVLC:master Mar 4, 2015
1 check passed

BestSonny commented on `d8c6aeb` Mar 7, 2015

dbcam commented on `include/caffe/blob.hpp` in `1434e87` Jan 14, 2016

seanbell commented on `src/caffe/blob.cpp` in `1434e87` Jan 16, 2016