Performance on MSMT17 is lower than the paper claims

[edwardnguyen1705]

Hi @Syliz517 ,
Thanks for sharing your great work.

I have downloaded some of pretrained model weights and evaluated them. On Market1501 and Duke, the results are the same as your paper claims, but it is not correct on MSMT17.
Regarding MSMT17, I downloaded the weights from ViT-CLIP-ReID-SIE-OLP and here is the config I used: msmt17.log. Please a look if I misconfig.
The result is here

=> MSMT17 loaded
Dataset statistics:
  ----------------------------------------
  subset   | # ids | # images | # cameras
  ----------------------------------------
  train    |  1041 |    32621 |        15
  query    |  3060 |    11659 |        15
  gallery  |  3060 |    82161 |        15
  ----------------------------------------
Resized position embedding: %s to %s torch.Size([197, 768]) torch.Size([211, 768])
Position embedding resize to height:21 width: 10
camera number is : 15
Loading pretrained model from checkpoints/MSMT17_clipreid_12x12sie_ViT-B-16_60.pth
2024-05-13 10:24:50,597 transreid.test INFO: Enter inferencing
The test feature is normalized
=> Computing DistMat with euclidean_distance
2024-05-13 10:30:45,340 transreid.test INFO: Validation Results 
2024-05-13 10:30:45,355 transreid.test INFO: mAP: 70.7%
2024-05-13 10:30:45,355 transreid.test INFO: CMC curve, Rank-1  :87.0%
2024-05-13 10:30:45,355 transreid.test INFO: CMC curve, Rank-5  :93.4%
2024-05-13 10:30:45,355 transreid.test INFO: CMC curve, Rank-10 :94.9%

Thanks for your time.

[chnbld]

Did you use re-ranking on test?

[edwardnguyen1705]

Hi @chnbld ,
I think they use H=384xW=128 resolution to test, but I use H=256xW=128.