The performance of MSMST

[yjh576]

Hello, I have a question about the performance of MSMT17. I find that the performance is related to the version of your code. The previous code is abbreviated as V1, and the current is abbreviated as V2. The perfrmace of V1 is the same as the performance of you paper, but the performace of V2 is worse than the performace of V1. I carefully investigate the code V1 and the code V2, bu I don't find the difference except for iters from 800 to 400. The above code is based on Kmeans method. When iters is set as 800 in V2, the performance is still worse than that of your paper.

Besides. I hace a question. That is, the time cost of V1 is much more than that of V2 for the Kmeans method. I find that this is because the clustering time except for iters and Jaccard distance. I want to know this is why?

Thank you!

[yxgeee]

You tried iters of 800 and kmeans clustering on V2, but the performance is still worse than V1?

[yjh576]

yes

[yxgeee]

That’s really strange. I did not change anything except for the iters. Is it possible that there exists some randomness?

[yjh576]

I try iters of 800 and DBSCAN clustering on V2, the performance is still worse than V1 with Kmeans. However, I try kmeans on V1 and its perforamnce is good. I am surprised. The time cost of V1 is much more than that of V2 for the Kmeans method. This is why?

[yxgeee]

DBSCAN is fast than K-Means, but not exactly better than K-Means. So you did not try iter800 and K-Means on V2? If you compare between V1+iter800+KMeans and V2+iter800+DBSCAN, the performance gap is normal.

[yxgeee]

You mean V2+KMeans is also faster than V1+KMeans?

[yjh576]

yes

[yxgeee]

Are you sure you run the train_mmt_kmeans.sh instead of train_baseline_kmeans.sh in V2?

[yxgeee]

Plus, did you run python setup.py install every time after changing the code?

[yxgeee]

If you store both two versions of code in you computer, you’d better run python setup.py install every time before running the code. Since two versions’ libraries mmt/ share the same name.

[yxgeee]

If you still cannot solve the problem, tell me your performance on V2+iter800+Kmeans. I will try it later when I am free.

[yjh576]

Fisrt, I confirmed it again.
I am sure that I run the train_mmt_kmeans.sh and don't run python setup,py. The two versions of code are placed in different folders. You can try it. I am interested in this . After that, I hope that we can have a further disscussion. Thank you!
I find that V2 has some simple change, but this is essentially unchanged. This don't influence the result. For example,
moving_avg_features = moving_avg_featuresargs.moving_avg_momentum+cf(1-args.moving_avg_momentum)
moving_avg_features = moving_avg_features / (1-args.moving_avg_momentum**(nc+1))

[yxgeee]

The two versions of code are placed in different folders.

This code does not import mmt directly from the current folder. It use python setup.py install to install the mmt library in your python directory. And then, it imports any functions in mmt from your python directory. Pls make sure you did not have a bug on this part.

[yjh576]

The result of V2+iter800+Kmeans is 14.2 (map), 34.2(top-1) | 46.6(top5) | 52.6(top10) .

[yxgeee]

The result of V2+iter800+Kmeans is 14.2 (map), 34.2(top-1) | 46.6(top5) | 52.6(top10) .

What is the number of clusters in kmeans?

[yjh576]

500

[yxgeee]

market2msmt or duke2msmt？resnet50？

[yjh576]

sorry, market2msmt + resnet50+500.
You said that this code does not import mmt directly from the current folder. I am surprised. I change loss in trainer.py of mmt folder and print some result. I am sure that this is changed.

[yxgeee]

Ok, I will check the problem with kmeans.
Plus, change rho from 1.6e-3 to 0.7e-3, DBSCAN-based MMT will achieve better performance on MSMT.
https://github.com/yxgeee/MMT/blob/master/examples/mmt_train_dbscan.py#L180

[yxgeee]

Ok, I will check the problem with kmeans.
Plus, change rho from 1.6e-3 to 0.7e-3, DBSCAN-based MMT will achieve better performance on MSMT.
https://github.com/yxgeee/MMT/blob/master/examples/mmt_train_dbscan.py#L180

I have tried rho=0.7e-3 with iters=400, achieving better performance than my paper.

[yjh576]

Ok, thank you for your good job. I hope that a further disscussion.

[yxgeee]

Hi,

I have run V2+Kmeans-500+iters400 for 30 epochs on Market-to-MSMT, and have already achieved similar performance as reported in the paper, i.e. mAP 16.7% now and 16.6% in the paper. I will upload my log.txt when finishing 40 epochs if you need it. Please make sure you did not make any modifications to the code.

[yxgeee]

Do you change the number of GPUs or the batch size for training? They may affect performance.

[yjh576]

Thank you. You needn't upload log.txt. I will check my code carefully, re-download the code, and run again.

[yxgeee]

Hi,
I have finished the training on V2+Kmeans-500+iters400 on Market-to-MSMT, the results are inspiring:

Mean AP: 18.1%
CMC Scores:
  top-1          40.5%
  top-5          53.3%
  top-10         59.5%