Improve TransposedConvolution layer. #1493

akhandait · 2018-08-13T20:14:02Z

No description provided.

src/mlpack/methods/ann/layer/transposed_convolution.hpp

ShikharJ · 2018-08-14T12:35:23Z

src/mlpack/tests/ann_layer_test.cpp


-  TransposedConvolution<> module2(1, 1, 4, 4, 1, 1, 2, 2, 5, 5);
+  TransposedConvolution<> module2(1, 1, 3, 3, 1, 1, 1, 1, 6, 6, 6, 6);


Please keep the original parameters here. These tests follow from the same Convolutional Arithmetic report, figure 4.1 and onwards.

@ShikharJ As far as I could figure out, tf.nn.conv2d_transpose doesn't allow size to go down(As it only allows VALID or SAME padding on the output size). In the above example, using the current formula for output size, it should be 4. Can you tell me what parameters(in the tf.nn.conv2d_transpose function) along with the below weights did you use to arrive at the value 2100?

@akhandait I don't think the size should decrease, take a look at Figure 4.2, aren't you able to obtain an output size of 8? I don't remember the parameters from the top of my head, though I can derive them, just give me sometime, I need to reinstall tensorflow. I have sort of moved on to Pytorch these days.

Sorry for the super late response. I understand the approach that was used. The output size calculated this way includes the padding added to the input of the equivalent convolution operation. However, most of the deep learning frameworks don't do that.(I tried Tensorflow and PyTorch. Thenao's approach is exactly the same as our new implementation.)
So, we will have to change the parameters. (but not the cases according to the paper)
For example, in this case, the output size should actually be 6(as shown in the figure, 1 + 1 is the padding and 8 is not the size of the original input. I tested it in PyTorch.) In PyTorch and our new implementation, this case should have parameters like this:

kernel size = 4 stride = 1 padding = 1(what all deep learning frameworks do is take the padding of the equivalent convolution operation, so here padding of 1 was used which changed the actual input size 6x6 to 8x8. We should output 6 and not 8.) inputHeight and Width = 5 outputHight and Width = 6

So, 5 = (6 + 2(1) - 4) / 1 + 1
But, as you correctly mentioned, we should test all the cases from that paper. I am going to shift to PyTorch for calculating the values for testing as Tensorflow sadly only allows 'SAME' and VALID padding which doesn't allow us to test all the cases from that paper.

@zoq Please verify this.

@ShikharJ @zoq Can you please comment on this?

@akhandait Sorry for the slow response, I don't see any reason against the change; I did also check PyTorch and as you already said it's the same parameter set.

@zoq Do you have any further comments on the PR? Can you give it a final review?

@akhandait Your approach sounds reasonable to me, I was actually referring to the decreased kernel sizes which have now been fixed. I'll need to run the tests for the GANs and test their output, and cross check the tests as well. I'm planning on doing these over the weekend. I'll let you know what I find.

@ShikharJ Sounds good. Thanks!

src/mlpack/tests/ann_layer_test.cpp

ShikharJ · 2018-08-15T17:02:14Z

@akhandait You can rebase to master now. #1491 is merged.

zoq · 2018-09-27T19:14:13Z

Looks like the Transposed layer test failed.

akhandait · 2018-10-01T10:06:15Z

@zoq It should pass now.

ShikharJ · 2018-10-01T12:27:50Z

src/mlpack/tests/ann_layer_test.cpp


-  TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 5, 5);
+  TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 2, 2, 5, 5);


Can you use the same input sizes here as well from the paper as done above? Also for module 6 and 7.

Maybe there's been some confusion. They already are according to the paper. The above module 5 has input size 2 and output size 5 which follows 4.5 in the paper, similarly the modules 6 and 7 follow 4.6 and 4.7 respectively.

ShikharJ · 2018-10-01T17:19:27Z

@akhandait Sorry if I'm being unreasonable, but I'm unable to understand the rationale behind inserting the zeros on the input first and then applying convolution operation. Could you explain a bit on that? Also, I think since both the ForwardConvolutionRule and BackwardConvolutionRule are valid, I think it'd be better to use them in Forward and Backward functions respectively.

akhandait · 2018-11-07T05:50:45Z

@zoq @ShikharJ Sorry for the inactivity on this one, I got busy with some other things. I will get back on this in a couple of days.

akhandait · 2018-11-11T06:08:11Z

@ShikharJ Inserting zeros has been done to implement fractionally strided convolutions with the current convolution rules. Another way could have been to implement another convolution rule for fractionally strided convolution which performs it without actually adding zeros in the input. But this way for easier for the time being.

akhandait · 2019-01-04T06:32:15Z

@ShikharJ Any updates here?

ShikharJ · 2019-01-04T08:07:46Z

@akhandait Apologies, I'll get back on reviewing this and give you an update soon.

zoq · 2019-03-30T18:49:08Z

@akhandait looks like there is a merge conflict, I can resolve this one if you like, just let me know. @ShikharJ Any update on this one from your side?

ShikharJ · 2019-03-30T19:17:54Z

@zoq I had been primarily busy with CycleGAN recently, but I think this is close to completion, just need to check the output on the GANs.

akhandait · 2019-03-31T11:07:07Z

I resolved it, let's check the tests now.

ShikharJ · 2019-04-06T07:42:05Z

@akhandait This seems to have a bit of style issues, do you think you can rectify them?

akhandait · 2019-04-10T08:27:18Z

Hey @ShikharJ, Jenkins isn't showing me where the checks have failed, it instead shows a Stack trace. Can you please restart the Jenkins build as I am not sure how to?

zoq · 2019-05-13T19:56:19Z

@mlpack-jenkins test this please

akhandait · 2019-05-25T14:45:07Z

@zoq @ShikharJ I can't understand the static checks warning here:
"The expression kW - padW - 1 > 0 will work as kW - padW != 1." - line 211, 233
Why is it != 1?
If they had said > 1, I wouldn't have worried about it, but I can't figure out this case.

rcurtin · 2019-05-25T19:54:07Z

@akhandait I think both kW and padW are size_t, so this has to do with the fact that negative numbers can't be represented. If kW < padW (which hopefully should not be the case) then kW - padW is some giant number greater than 0. So, the only situation for which we can make the condition kW - padW - 1 > 0 false is when kW - padW - 1 == 0, or, when kW - padW == 1 (hence, the suggested condition of kW - padW != 1. I think this is right, hope it helps. :)

akhandait · 2019-05-26T10:07:03Z

@rcurtin Ahh I missed that, silly error. I will just cast it to int.

saksham189 · 2019-07-01T16:03:59Z

src/mlpack/methods/ann/layer/transposed_convolution.hpp

@@ -222,18 +214,23 @@ class TransposedConvolution
   * @param input The input to be padded.
   * @param wPad Padding width of the input.
   * @param hPad Padding height of the input.
+   * @param wExtra The number of extra zeros to the right.
+   * @param hExtra The number of extra zeros to the bottom.
   * @param output The padded output data.
   */
  template<typename eT>
  void Pad(const arma::Mat<eT>& input,


Maybe we could introduce a Padding layer and use that inside the convolution layers to avoid redundant code. Pytorch already has support for quite a lot of padding layers. This would also make the implementation more clear.

@saksham189 Has there been any discussion on this on irc after you posted this? I think this might be good idea and should be given some thought.

Sorry for the slow response, makes totally sense to me.

@akhandait I will work on adding the Padding layer and remove the redundant code in a new PR so that it is easy to review.

Looks like the Padding layer is done and merged; should we incorporate those changes here, or maybe in another PR after this is merged? It may be worth opening an issue so we don't forget. :)

mlpack-bot · 2019-08-04T21:25:44Z

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍

akhandait · 2019-09-15T15:10:43Z

@rcurtin I guess that would be much better for me right now, thanks! I'll push the changes tomorrow.

toshal-a

Hi, it looks good. However, I found one small mistake. It's quite small. But maybe quite important.

toshal-a · 2019-09-24T12:14:15Z

src/mlpack/methods/ann/layer/transposed_convolution_impl.hpp

    weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
+
+    aW = (outputWidth + kW - 2 * padW - 2) % dW;


If we are serializing these parameters then we don't need to calculate them while loading. Let me know if it's not correct. May be I missing something.

Ah, actually, this is a good point. If we are serializing aW and aH, then we have to handle reverse compatibility. (That's not hard but it's just a little bit tedious.) Since as far as I can tell aW and aH are always computed, I'd suggest that we just remove the two lines ar & BOOST_SERIALIZATION_NVP(aw); and ar & BOOST_SERIALIZATION_NVP(aH); above.

rcurtin · 2019-09-26T01:38:56Z

@walragatver pointed out in IRC today that we shouldn't wait on the release any longer, as mlpack 3.1.1 downloads ensmallen-latest.tar.gz---which as of ensmallen 2.10.0 no longer works! I agreed, so I've gone ahead and released 3.2.0 tonight (which doesn't have the same issue as 3.1.1 did with respect to the ensmallen version), and I've moved this to the 3.2.1 milestone. Once this is finished and merged we can easily issue a patch release. 👍

rcurtin

I don't have any further comments on this one other than about the serialization; do we think that it's ready otherwise? I'd love to get this merged and release 3.2.2 soon. :)

rcurtin · 2019-10-06T21:52:19Z

src/mlpack/methods/ann/layer/transposed_convolution_impl.hpp

    weights.set_size((outSize * inSize * kW * kH) + outSize, 1);
+
+    aW = (outputWidth + kW - 2 * padW - 2) % dW;


Ah, actually, this is a good point. If we are serializing aW and aH, then we have to handle reverse compatibility. (That's not hard but it's just a little bit tedious.) Since as far as I can tell aW and aH are always computed, I'd suggest that we just remove the two lines ar & BOOST_SERIALIZATION_NVP(aw); and ar & BOOST_SERIALIZATION_NVP(aH); above.

rcurtin · 2019-10-06T22:52:28Z

It looks like there is still a travis test failing, also. If nobody has any time, I can try and debug it.

akhandait · 2019-10-07T07:08:52Z

Sorry for stalling here. I am having issues building mlpack on my system, so it became harder to work on this as I need to wait for travis to build and then debug through those errors. I'll still try completing this today.

zoq · 2019-10-07T20:17:27Z

Hopefully the build issues you see, are fixed with the latest ensmallen + mlpack version.

rcurtin · 2019-10-09T02:09:33Z

@akhandait no problem, and I'm happy to help try and debug your system if you like, just let me know. I think @zoq is right that the update of ensmallen might be the thing that makes a difference here; if you make sure the branch is rebased against master and then delete/remake your build directory, that should work out any issues (I hope).

akhandait · 2019-10-09T18:22:59Z

@zoq It didn't :(. I am using ensmallen 2.10.3 and mlpack latest master(so rebase isn't an issue). I just posted the entire error on irc. Can you please see if it seems familiar? If not I will post the build config so that someone can reproduce it.

akhandait · 2019-10-10T13:40:07Z

Hmm, one of the jobs in travis is still failing due to some reason. I don't think it's due to this PR though.

rcurtin · 2019-10-10T14:02:59Z

You're right, it just timed out. Let me try running it again and we'll see what happens...

rcurtin

Everything looks good to me and the tests are passing. 👍 There was one comment about using the Padding layer, so I think when we merge this we should be sure to open an issue about that:

https://github.com/mlpack/mlpack/pull/1493/files#r299114822

Also, either before merge or during merge, we should update HISTORY.md also. :)

mlpack-bot

Second approval provided automatically after 24 hours. 👍

akhandait · 2019-10-12T03:27:30Z

I forgot to update HISTORY.md before the merge, could someone please do it? Or should I commit directly to the repository?

zoq · 2019-10-12T19:24:42Z

Updated HISTORY.mdin 7d677d0.

ShikharJ reviewed Aug 14, 2018

View reviewed changes

src/mlpack/methods/ann/layer/transposed_convolution.hpp Outdated Show resolved Hide resolved

ShikharJ reviewed Aug 14, 2018

View reviewed changes

src/mlpack/tests/ann_layer_test.cpp Outdated Show resolved Hide resolved

akhandait force-pushed the transposed_conv_fix branch 2 times, most recently from 41654d0 to a3e5c2f Compare August 29, 2018 14:29

ShikharJ reviewed Oct 1, 2018

View reviewed changes

akhandait force-pushed the transposed_conv_fix branch from 4cad362 to 7974bdc Compare November 26, 2018 06:29

rcurtin added c: methods t: added feature labels Jan 19, 2019

akhandait force-pushed the transposed_conv_fix branch from 6c58629 to f812835 Compare May 25, 2019 12:15

akhandait force-pushed the transposed_conv_fix branch from f812835 to b468374 Compare May 26, 2019 10:05

saksham189 reviewed Jul 1, 2019

View reviewed changes

toshal-a reviewed Sep 24, 2019

View reviewed changes

rcurtin modified the milestones: mlpack 3.2.0, mlpack 3.2.0 (maybe), mlpack 3.2.1, mlpack 3.2.2 Sep 26, 2019

rcurtin reviewed Oct 6, 2019

View reviewed changes

akhandait added 5 commits October 9, 2019 23:43

Improve TransposedConvolution layer.

0f6acfd

Correct Gradient() function.

86cf165

Use corresponding typenames for Forward and Backward.

d76d388

Fix style and static code analysis issues.

f00b1bb

Make suggested changes.

61fcdeb

Fix failing tests.

d2c78df

akhandait force-pushed the transposed_conv_fix branch from 48667d6 to d2c78df Compare October 10, 2019 08:15

rcurtin approved these changes Oct 10, 2019

View reviewed changes

mlpack-bot bot approved these changes Oct 11, 2019

View reviewed changes

akhandait merged commit 44457ef into mlpack:master Oct 12, 2019

akhandait mentioned this pull request Oct 12, 2019

Use Padding layer in TransposedConvolution layer #2051

Closed

ShikharJ added this to To Review in Shikhar's Board Oct 13, 2019

kartikdutt18 mentioned this pull request May 31, 2020

Bug Fix : Check correctness of shape in Transposed Conv. layer only when output width and height aren't zero. #2436

Merged


		TransposedConvolution<> module2(1, 1, 4, 4, 1, 1, 2, 2, 5, 5);
		TransposedConvolution<> module2(1, 1, 3, 3, 1, 1, 1, 1, 6, 6, 6, 6);


		TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 5, 5);
		TransposedConvolution<> module5(1, 1, 3, 3, 2, 2, 0, 0, 2, 2, 5, 5);

		weights.set_size((outSize * inSize * kW * kH) + outSize, 1);

		aW = (outputWidth + kW - 2 * padW - 2) % dW;

Improve TransposedConvolution layer. #1493

Improve TransposedConvolution layer. #1493

Conversation

akhandait commented Aug 13, 2018

ShikharJ Aug 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhandait Aug 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShikharJ commented Aug 15, 2018

zoq commented Sep 27, 2018

akhandait commented Oct 1, 2018

ShikharJ Oct 1, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShikharJ commented Oct 1, 2018

akhandait commented Nov 7, 2018

akhandait commented Nov 11, 2018

akhandait commented Jan 4, 2019

ShikharJ commented Jan 4, 2019

zoq commented Mar 30, 2019

ShikharJ commented Mar 30, 2019

akhandait commented Mar 31, 2019

ShikharJ commented Apr 6, 2019

akhandait commented Apr 10, 2019

zoq commented May 13, 2019

akhandait commented May 25, 2019

rcurtin commented May 25, 2019

akhandait commented May 26, 2019

saksham189 Jul 1, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlpack-bot bot commented Aug 4, 2019

akhandait commented Sep 15, 2019

toshal-a left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcurtin commented Sep 26, 2019

rcurtin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcurtin commented Oct 6, 2019

akhandait commented Oct 7, 2019

zoq commented Oct 7, 2019

rcurtin commented Oct 9, 2019

akhandait commented Oct 9, 2019

akhandait commented Oct 10, 2019

rcurtin commented Oct 10, 2019

rcurtin left a comment

Choose a reason for hiding this comment

mlpack-bot bot left a comment

Choose a reason for hiding this comment

akhandait commented Oct 12, 2019

zoq commented Oct 12, 2019 • edited

ShikharJ Aug 14, 2018 •

edited

akhandait Aug 29, 2018 •

edited

ShikharJ Oct 1, 2018 •

edited

saksham189 Jul 1, 2019 •

edited

zoq commented Oct 12, 2019 •

edited