Having troble reproducing "ours-FC"

[wishforgood]

Following the decribtion in the paper, I add a new ODE function:
class odeFunc3(nn.Module):

def __init__(self, use_cuda):
    super(odeFunc3, self).__init__()
    self.use_cuda = use_cuda
    self.linear = nn.Linear(1024, 1024)
    self.hidden_state = None

def init_hidden(self, batch_size):
    return None

def forward(self, t, x):
    x = self.linear(x)
    return x

and replace the two odeFunc used in NODIS.py with it. Then I train the PREDCLS model with:
python models/train_rels.py -m predcls -order random -b 6 -p 100 -lr 1e-4 -ngpu 1 -ckpt checkpoints/vgdet/vg-24.tar -save_dir checkpoints/ -nepoch 20
But when I test the resulting model with:
python models/eval_rels.py -m predcls -order random -b 6 -p 100 -lr 1e-3 -ngpu 1 -test -ckpt checkpoints/vgrel-19-0.6350.tar -nepoch 50
I only got 0.658502 on R@100 for the PREDCLS task.
Did I miss anything?

[wishforgood]

Oh, I will try to train SGCLS model based on this model and test it again.

[wishforgood]

I'm still not able to reproduce the result, I got 0.67 on R@100 for the PREDCLS task using the trained SGCLS model.

[yrcong]

Hi,
you should not replace the Bi-LSTM in ode block by Linear layer. But you can replace the LSTM before ODE-block. Btw, how about R@50?

[wishforgood]

Sorry, I'm still not clear why the Bi-LSTM in ode block cannot be replaced by Linear layer. I thought this is how the paper describes "ours-FC":

(1) One in which the function used inside the ODE layer is taken to be a linear layer (ours-FC)

Apparently both the version using a linear layer inside the ODE layer (ours-FC) and the one using the bidirectional LSTM (ours-LSTM) perform very similar.

It is sufficient to use a simple linear layer, since the ODE layer is so powerful that it produces similar outputs to those of a function based upon a more complicated and slower bidirectional LSTM.

I guess by " replace the LSTM before ODE-block", you mean:

The visual vectors and semantic vectors of object pairs can be pre-processed by three methods before the P-ODE: FC-Layer: The (3 · 512)-dimensional visual vectors and 400-dimensional semantic vectors are forwarded into two independent fully connection layers that both have 512 neurons. Then, the outputs are concatenated together as 1024-dimensional representation vectors for the P-ODE.

I failed to record the result on R@50 because I thought it was not a satisfying result, but I can reproduce it soon (in about 10 minutes).

[yrcong]

ohh, i got you.. That part is written by another author, he misunderstood me. ours-FC means using FC before ODE-block. Then different network structures will provide different representations for the ODE block.

[wishforgood]

OK, thank you, that makes sense. I will experiment with that setting to see if I can reproduce the result. Btw, the R@50 was 0.648, R@20 was 0.564.

[yrcong]

Are these scores (Re@50=0.648) using FC in ODE-block?

[wishforgood]

Yes

[wishforgood]

And this the SGCLS results using FC in ODE-block.

[yrcong]

Ok that is reasonable, thanks

[wishforgood]

Hi, as you suggested, I change the LSTMs before the ode block to FC-layers for reproducing "ours-FC", I defined them as follows:

    self.linear_visual = nn.Linear(1536, 512)

    self.linear_semantic = nn.Linear(400, 512)

    self.odeBlock = odeBlock(odeFunc1(bidirectional=True))

And applied them in the forward function of NODIS class:

    perm, inv_perm, ls_transposed = self.sort_rois(rel_inds[:, 0].data, None, union_rois[:, 1:])

    visual_input = PackedSequence(x_visual[perm], torch.tensor(ls_transposed))

    inputs1, lengths1 = pad_packed_sequence(visual_input, batch_first=False)

    semantic_input = PackedSequence(x_semantic[perm], torch.tensor(ls_transposed))

    inputs2, lengths2 = pad_packed_sequence(semantic_input, batch_first=False)   

    output1 = self.linear_visual(inputs1)

    output2 = self.linear_semantic(inputs2)       

    inputs = torch.cat((output1, output2), 2)

    x_fusion = self.odeBlock(inputs, batch_size)

I didn't remove the functions PackedSequence and pad_packed_sequence (I'm not familiar with RNN, but I guess it will not affect how the outputs are computed). Other parts are kept unchanged. The ODE functions are kept as LSTMs.
The results are as follows:

We can see that the performance of SGCLS is still far from reported performance. Did I miss any thing?

[wishforgood]

But I got a better-than-reported performance with only 6 epochs training of SGDET:

[yrcong]

That's a good news! Did you use our object detector?

[wishforgood]

Sure, I got an even better result (R@50: 0.280 R@100: 0.314) later on epoch 7. Is there any difference between my implementation and the correct one? Though there seems to be no big problem with the other two tasks, the performance on SGCLS decreases significantly.