fail to reproduce

[IQ17]

Hi, I am reproducing the work. Following the readme, I train the resnet101 with default settings except for a little path changes.

However, the evaluation results on ROxford(no 1M distractors) is too poor, and I get mAP_E < 70, mAP_M < 50 and mAP_H < 20。Similar things on the RParis evaluation.

[justforforfor]

Have you trained the network for four rounds? And each round should load the model trained on the previous round excepted that for the first round, ImageNet-pretrained should be loaded.

[IQ17]

I have done the 3rd stage and used the weights from the previous stage for sure. Yet, the results of the 3rd stage remained poor (mAP_E < 70, mAP_M < 50 and mAP_H < 20 on ROxford). I am training the 4th stage now, with no clear improvement at the moment.

[IQ17]

Hi, I just finished stage 4 for GLDv2, with resnet101+GeM(p=3)+Head(norm_linear_norm) as given in the configs.

The reproduced results are here:

• Test: img_size=1024 scales=[1.0, 1/sqrt(2), 0.5]
  ROxford: M: 64.12 H: 37.83;
  RParis: M: 74.54 H: 50.91, and the best results are M: 74.97, H: 51.37

• Test: img_size=1024, scales=[sqrt(2), 1.0, 1/sqrt(2) ]
  ROxford: M: 63.48 H: 38.47;
  RParis: M: 72.1 H: 47.26

The results taken from the Table 3 are much better, so what maybe wrong ?

• Test: img_size=1024, scales=[sqrt(2), 1.0, 1/sqrt(2) ]
  ROxford: M: 73.1 H: 48.3;
  RParis: M: 77.6 H: 59.5

[justforforfor]

Sorry for the late replay. I believe the problem should be caused by the first three rounds of training (base stages). You can download the trained model (after three rounds of training) and start the 4th training (use the impr config), then you should reach the performance reported in the paper. Checkpoint can be downloaded here.

Note that you may need to change some parameter name to load the checkpoint.

[IQ17]

Hi, thanks for the checkpoint, and I am going to have a try.

I notice that this checkpoint contains keys like "combiner.spatial_attention.conv1.weight", which belong to the spatial attention module. In this git repository, the spatial_attention is off in both config and forward function. Does it help in your experiments?

[IQ17]

Hi, I have done the 4th stage with your ckpt, and the results are similar to the ones on paper.

• Test: img_size=1024 scales=[1.0, 1/sqrt(2), 0.5]
  ROxford: M: 70.98 H: 44.91;
  RParis: M: 78.48 H: 61.02;

How do you get this ckpt of the 3rd stage? Anything I missed?

Thanks!

[peternara]

Thanks for sharing your code. @zeludeng

I am also reproducing with your code (all stages..).
However, it is far below the performance of your paper. (About 5-10% of all items are low.) similar...@IQ17
(I did it without changing any parameters in the code.)

Also, the evaluation using the trained model (s3_pretrained.pth) you shared above is very low.
I used your code (https://github.com/zeludeng/insclr/blob/main/tools/test_net.py) as it is for the evaluation code after loading your model.
Below are the evaluation results(s3_pretrained.pth). What's the problem? @zeludeng @IQ17

CUDA_VISIBLE_DEVICES=2 python tools/test_net.py --cfg configs/gldv2/impr.yaml

2022-05-14 15:12:34,341 insclr INFO: loading pretrained backbone from insclr_pretrained/s3_pretrained.pth
2022-05-14 15:12:34,523 insclr INFO: missing keys: []
2022-05-14 15:12:34,523 insclr INFO: unexpected keys: ['combiner.spatial_attention.conv1.weight', 'combiner.spatial_attention.conv1.bias', 'combiner.spatial_attention.conv2.weight', 'combiner.spatial_attention.conv2.bias', 'combiner.spatial_attention.reduce.weight', 'combiner.spatial_attention.reduce.bias', 'combiner.attention_head.fc.weight']
2022-05-14 15:12:34,538 insclr INFO: Sequential(
  (backbone): ResNet(
    (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    .....
    
    
2022-05-14 15:12:36,967 insclr INFO: evaluator: dataset=roxford5k scales=[1, 0.7071067811865476, 0.5] imsize=1024
2022-05-14 15:12:36,967 insclr INFO: evaluator: dataset=rparis6k scales=[1, 0.7071067811865476, 0.5] imsize=1024
  0%|                                                                                                                                                   | 0/70 [00:00<?, ?it/s]/home/cheolhwan/.conda/envs/cheolhwan_ir/lib/python3.8/site-packages/torch/nn/functional.py:3502: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
  warnings.warn(
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 70/70 [00:05<00:00, 13.51it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4993/4993 [06:12<00:00, 13.39it/s]
2022-05-14 15:18:55,265 insclr INFO: 
>> roxford5k: mAP E: 71.53, M: 56.21, H: 30.81

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 70/70 [00:04<00:00, 14.07it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6322/6322 [07:51<00:00, 13.41it/s]
2022-05-14 15:26:52,213 insclr INFO: 
>> rparis6k: mAP E: 84.05, M: 65.87, H: 38.91

[IQ17]

Hi, @peternara ! I think I can answer your 2nd question.
The ckpt provided by @zeludeng is only the result of stage 3, not the final model.
As he suggested(see his comments above), I did the 4th stage with his ckpt, and the results are on par with the ones on paper.

But I also failed to reproduce from scratch myself, and the reason is unclear.

Hi, I have done the 4th stage with your ckpt, and the results are similar to the ones on paper.

• Test: img_size=1024 scales=[1.0, 1/sqrt(2), 0.5]
  ROxford: M: 70.98 H: 44.91;
  RParis: M: 78.48 H: 61.02;

How do you get this ckpt of the 3rd stage? Anything I missed?

Thanks!

[peternara]

Ooh I didn't read carefully.
thank you. @IQ17
I will try it.

However, I can't understand why it can't be reproduced with this published code.
In the last stage training, I saw a performance increase over the previous one. However, this performance result is much lower than that of paper. weird..