PyramidROIAlign shape question

[happynewya]

I have a question when I read codes of PyramidROIAlign class in model.py. The first dimension of pooled should be batch, but I didnt get a right result.
I constructed test inputs:

boxes = np.array([
    [
        [0.1, 0.1, 0.3, 0.5],
        [0.1, 0.2, 0.65, 0.9],
        [0.12, 0.32, 0.5, 0.4]
    ],
    [
        [0.12, 0.12, 0.3,   0.53],
        [0.12, 0.23, 0.365, 0.49],
        [0.1,  0.6,  0.4, 0.9]
    ]
], dtype=np.float32)
image_meta = compose_image_meta(123, [512,512,3], [1024,1024,3], (0,0,1023,1023), 2, [1,2,3])
image_meta = np.stack((image_meta,image_meta), axis=0)
P2 = np.random.rand(32, 32, 64)
P2 = np.stack((P2, P2))
P2 =K.variable(P2)
P3 = P2
P4 = P2
P5 = P2
feature_maps = [P2, P3, P4, P5]
inputs = [K.variable(boxes), K.variable(image_meta)] + feature_maps

and feed them into PyramidROIAlign Layer

prl = PyramidROIAlign([7, 7], name="roi_align_classifier")
x = prl(inputs)

I checked the outputs shape and found mismatch between x.shape and compute_output_shape()

x.shape
>>>(1, 6, 7, 7, 64)
prl.compute_output_shape(prl.input_shape)
>>>(2, 3, 7, 7, 64)

I reviewed the code inside PyramidROIAlign and found this. I thought it didn't "re-add" the batch dimension information back to pooled

 # Re-add the batch dimension
        pooled = tf.expand_dims(pooled, 0)

Should this code be modified to code below?

pooled = tf.reshape(pooled, (batchsize, boxesnum_per_img) + pooled.shape[2:])

[happynewya]

I think this is a bug. But due to the bbox loss computation order, it wont cause exception or miscalculation.
To be confirmed after reading the code

[hnyz979]

It is very strange that the x is a 5-dimensional tensor. It seems that this is caused by the KL.distributed function by the KERAS engine. I am not familiar with KERAS, so I changed all these to tensorflow. The first two dimensions are merged.

[hejujie]

I use this function in tensorflow, but I get the output with shape (?, 7, 7, 64), I replace the expand with reshape by pooled = tf.reshape(pooled, ([boxes.shape[0], boxes.shape[1]] + list(pooled.shape[1:]))), and get the output with shape (2, 200, 7, 7, 64)， where 2 is the batch size, 200 is the num_roi.

[waleedka]

@zhydong I think you're right. Good catch! Instead of returning a shape [batch, boxes, H, W, C], the PyramidROIAlign is returning [1, batch*boxes, H, W, C].

However, the output from this layer is immediately fed to a TimeDistributed layer, which in effect merge the first two dimensions (batch & boxes), then apply a convolution, and then separate the batch and boxes dimensions. And it's separating the dimensions correctly, which corrects the bug in the earlier line.

So, as far as I can tell, yes this is a bug, but it's not causing any issues in training or detection. I think it's better to fix it, though. If you submit a PR I'd be happy to merge it. Alternatively, I'll try to make time to fix and test it.

[waleedka]

I pushed a fix for this, so I'm closing this issue.