add bilinear upsample layer #1136

kczimm · 2020-04-21T19:45:31Z

Implement a 2D bilinear upsample layer like the one found in tensorflow.

johnnychen94

need docs
need test

src/layers/upsample.jl

johnnychen94 · 2020-04-21T19:54:40Z

src/layers/upsample.jl

+
+    out = similar(x, (newW, newH, C, N))
+
+    for n = 1:N, c = 1:C, w = 1:newW, h = 1:newH


This scalar loop would be extremely slow when x is CuArray

Agreed, but I must admit I don't have experience optimizing Julia for CUDA usage. Do you have suggestions in this case?

Broadcast would be fast on GPU

kczimm · 2020-04-23T14:16:21Z

build failed perhaps due to the Github outage I was experiencing? HTTP 500 errors on clones. I don't know how to trigger a rebuild without pushing another commit.

johnnychen94

The implementation looks good to me, although I'm not sure if this is the optimized one wrt CuArrays. Interpolation could be interpreted as a convolution operation, so it might be possible to have a more efficient implementation, which I'm not sure of. It would be better if someone familiar with this topic can help review.

BTW, can you also add this layer to the docs? I think it could be https://github.com/FluxML/Flux.jl/blob/master/docs/src/models/layers.md#L10

johnnychen94 · 2020-04-23T23:46:46Z

src/layers/upsample.jl

+ 0.823648  0.658877  0.329336  0.164566
+ 0.845325  0.675933  0.337149  0.167757
+ 0.888679  0.710044  0.352773  0.174138
+ 0.910357  0.7271    0.360586  0.177329```


- 0.910357 0.7271 0.360586 0.177329``` + 0.910357 0.7271 0.360586 0.177329 + ```

johnnychen94 · 2020-04-23T23:50:14Z

src/layers/upsample.jl

+
+Create an upsampling layer that uses bilinear interpolation.
+
+The width and height dimensions grow by the `factor` tuple.


The 1st dimension is height because in Julia it's column first.

Suggested change

The width and height dimensions grow by the `factor` tuple.

The input is commonly interpreted as a batch of images, where the height(1st) and width(2nd) dimensions grow by the `factor` tuple.

johnnychen94 · 2020-04-23T23:51:54Z

src/layers/upsample.jl

+"""
+    BilinearUpsample(factor::Tuple{Integer,Integer})
+
+Create an upsampling layer that uses bilinear interpolation.


Suggested change

Create an upsampling layer that uses bilinear interpolation.

Create an upsampling layer that uses bilinear interpolation to upscale the first two dimensions of 4D input.

johnnychen94 · 2020-04-23T23:52:31Z

src/layers/upsample.jl

+    factor::Tuple{T,T}
+end
+
+function (b::BilinearUpsample)(x::AbstractArray)


Suggested change

function (b::BilinearUpsample)(x::AbstractArray)

function (b::BilinearUpsample)(x::AbstractArray{T, 4}) where T

johnnychen94 · 2020-04-23T23:59:52Z

src/layers/upsample.jl

+end
+
+function (b::BilinearUpsample)(x::AbstractArray)
+    W, H, C, N = size(x)


It might be better to swap W and H but it's still okay to keep it as it is, since most people confuse them.

The data in Flux is stored in WHCN order isn’t it?

Flux.jl/src/layers/conv.jl

Line 17 in 7a32a70

Data should be stored in WHCN order (width, height, # channels, batch size).

It is, but I prefer to read it as a misunderstanding. I just wanted to mention it here in case you're not aware of it.

It's okay to abuse the usage of WH since it's relative to the column/row-first order.

kczimm · 2020-04-24T14:39:13Z

I put this in a new file which I was hesitant to do. Perhaps it belongs with the conv code and docs because this is really used in convolutional networks?

johnnychen94 · 2020-04-24T17:48:36Z

I put this in a new file which I was hesitant to do. Perhaps it belongs with the conv code and docs because this is really used in convolutional networks?

I guess where it should be put doesn't matter much. One thing I forgot to mention is to export the symbol BilinearUpsample in src/Flux.jl.

DhairyaLGandhi · 2020-04-24T19:25:05Z

Let's add gpu tests, and maybe we could think of performance on the GPU. It might be that we want to go with a broadcasting approach

scimas · 2020-05-01T02:39:38Z

Linking this code written for bilinear interpolation using Interpolations.jl for reference, in case it offers any value for optimizing this pr. comment

CarloLucibello · 2020-05-01T07:36:28Z

The approach in this PR seems neither Zygote nor Flux friendly, I don't think this is the way we go. @scimas did you test your code with differentiation and on gpu?

scimas · 2020-05-01T16:07:51Z

@CarloLucibello I just got around to testing it today, doesn't work with zygote. I get errors about being unable to check bounds.

scimas · 2020-05-02T01:22:01Z

So I did some more testing and Zygote calculates the gradient without any problem if I explicitly tell it to perform forwarddiff. Still have not tested on GPU.

using Zygote
up = Upsampler(2, 2);
x = rand(2, 2, 1, 1);
model(x) = sum(up(x));

# This works
grad = gradient(x) do p
    Zygote.forwarddiff(p) do p
        model(p)
    end
end

# But this doesn't
grad = gradient(x) do x
    model(x)
end

where Upsampler is from the previously linked code.

And frankly I don't have enough expertise to figure out why backward diff doesn't work, simply figuring out the forwarddiff part took most of the day.

kczimm · 2020-05-13T14:55:06Z

Closing in favor of the superior #1180. Thanks everyone who reviewed this PR.

add bilinear upsample layer

734afc5

johnnychen94 reviewed Apr 21, 2020

View reviewed changes

kczimm added 3 commits April 22, 2020 20:28

add doctest. add include.

2d0bd18

remove batch/channel loop. add @inbounds and @views.

d2fee6d

fixed doc sentence.

3110531

kczimm marked this pull request as draft April 23, 2020 12:02

add tests

0d9b84f

kczimm marked this pull request as ready for review April 23, 2020 13:27

fix grammer in doc

a15f02e

johnnychen94 approved these changes Apr 24, 2020

View reviewed changes

johnnychen94 mentioned this pull request May 1, 2020

imresize spuriously causes NaNs with 4d image JuliaImages/Images.jl#763

Open

ltjkoomen mentioned this pull request May 13, 2020

Add bilinear upsample layer #1180

Closed

kczimm closed this May 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add bilinear upsample layer #1136

add bilinear upsample layer #1136

kczimm commented Apr 21, 2020

johnnychen94 left a comment •

edited

Loading

johnnychen94 Apr 21, 2020

kczimm Apr 21, 2020

AriMKatz Apr 21, 2020

kczimm commented Apr 23, 2020

johnnychen94 left a comment •

edited

Loading

johnnychen94 Apr 23, 2020

johnnychen94 Apr 23, 2020

johnnychen94 Apr 23, 2020

johnnychen94 Apr 23, 2020

johnnychen94 Apr 23, 2020

kczimm Apr 24, 2020

johnnychen94 Apr 24, 2020 •

edited

Loading

kczimm commented Apr 24, 2020

johnnychen94 commented Apr 24, 2020 •

edited

Loading

DhairyaLGandhi commented Apr 24, 2020

scimas commented May 1, 2020

CarloLucibello commented May 1, 2020

scimas commented May 1, 2020

scimas commented May 2, 2020 •

edited

Loading

kczimm commented May 13, 2020


		out = similar(x, (newW, newH, C, N))

		for n = 1:N, c = 1:C, w = 1:newW, h = 1:newH


		Create an upsampling layer that uses bilinear interpolation.

		The width and height dimensions grow by the `factor` tuple.

	The width and height dimensions grow by the `factor` tuple.
	The input is commonly interpreted as a batch of images, where the height(1st) and width(2nd) dimensions grow by the `factor` tuple.

	function (b::BilinearUpsample)(x::AbstractArray)
	function (b::BilinearUpsample)(x::AbstractArray{T, 4}) where T

add bilinear upsample layer #1136

add bilinear upsample layer #1136

Conversation

kczimm commented Apr 21, 2020

johnnychen94 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kczimm commented Apr 23, 2020

johnnychen94 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnnychen94 Apr 24, 2020 • edited Loading

Choose a reason for hiding this comment

kczimm commented Apr 24, 2020

johnnychen94 commented Apr 24, 2020 • edited Loading

DhairyaLGandhi commented Apr 24, 2020

scimas commented May 1, 2020

CarloLucibello commented May 1, 2020

scimas commented May 1, 2020

scimas commented May 2, 2020 • edited Loading

kczimm commented May 13, 2020

johnnychen94 left a comment •

edited

Loading

johnnychen94 left a comment •

edited

Loading

johnnychen94 Apr 24, 2020 •

edited

Loading

johnnychen94 commented Apr 24, 2020 •

edited

Loading

scimas commented May 2, 2020 •

edited

Loading