Pre-train model Performance

[durbin-164]

Hi,
Thanks for this great work. I have tried to use pre-train model but get very low performance. Would you please help me to find out where I made mistake.

I ran with this command.
python world_track.py test -c model_weights/wild_segnet/config.yaml
--ckpt model_weights/wild_segnet/model-epoch=21-val_loss=7.79-val_center=4.76.ckpt

And got this values:
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
0 1.2% 31.6% 0.6% 0.6% 31.6% 41 0 0 41 13 946 0 0 -0.7% 0.566 0 0 0
OVERALL 1.2% 31.6% 0.6% 0.6% 31.6% 41 0 0 41 13 946 0 0 -0.7% 0.566 0 0 0
Testing DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:30<00:00, 1.30it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ detect/moda │ 0.0 │
│ detect/modp │ 34.748539923465074 │
│ detect/precision │ 4.545454545454546 │
│ detect/recall │ 0.20491803278688525 │
│ track/idf1 │ 1.2358393669128418 │
│ track/idp │ 31.578947067260742 │
│ track/idr │ 0.6302521228790283 │
│ track/mostly_lost │ 1.0 │
│ track/mostly_tracked │ 0.0 │
│ track/mota │ -0.7352941036224365 │
│ track/motp │ 43.35531234741211 │
│ track/num_ascend │ 0.0 │
│ track/num_false_positives │ 13.0 │
│ track/num_fragmentations │ 0.0 │
│ track/num_migrate │ 0.0 │
│ track/num_misses │ 946.0 │
│ track/num_switches │ 0.0 │
│ track/num_transfer │ 0.0 │
│ track/num_unique_objects │ 41.0 │
│ track/partially_tracked │ 0.0 │
│ track/precision │ 31.578947067260742 │
│ track/recall │ 0.6302521228790283 │
└───────────────────────────┴───────────────────────────┘

[tteepe]

Hi,

can you try again with batch size 1? Our caching implementation requires this.
world_track.py test -c ../model_weights/wild_segnet/config.yaml --ckpt ../model_weights/wild_segnet/checkpoints/model-epoch=21-val_loss=7.79-val_center=4.76.ckpt --data.batch_size 1

This gives me the following output:

│        detect/moda        │     92.1218487394958      │
│        detect/modp        │     76.20120505981272     │
│     detect/precision      │     96.9989281886388      │
│       detect/recall       │     95.06302521008404     │
│        track/idf1         │     95.30838012695312     │

Cheers,
Torben

[RashoAli]

Hi,
Unfortunatly i hade the exact same problem (with the same values).
i used the comand:

python world_track.py test -c ./model_weights/wild_segnet/config.yaml --ckpt ./model_weights/wild_segnet/model-epoch=21-val_loss=7.79-val_center=4.76.ckpt --data.batch_size 1

the results are:
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
0 1.2% 31.6% 0.6% 0.6% 31.6% 41 0 0 41 13 946 0 0 -0.7% 0.566 0 0 0
OVERALL 1.2% 31.6% 0.6% 0.6% 31.6% 41 0 0 41 13 946 0 0 -0.7% 0.566 0 0 0
Testing DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:24<00:00, 1.62it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ detect/moda │ 0.0 │
│ detect/modp │ 34.76547430153938 │
│ detect/precision │ 4.545454545454546 │
│ detect/recall │ 0.20491803278688525 │
│ track/idf1 │ 1.2358393669128418 │
│ track/idp │ 31.578947067260742 │
│ track/idr │ 0.6302521228790283 │
│ track/mostly_lost │ 1.0 │
│ track/mostly_tracked │ 0.0 │
│ track/mota │ -0.7352941036224365 │
│ track/motp │ 43.35794448852539 │
│ track/num_ascend │ 0.0 │
│ track/num_false_positives │ 13.0 │
│ track/num_fragmentations │ 0.0 │
│ track/num_migrate │ 0.0 │
│ track/num_misses │ 946.0 │
│ track/num_switches │ 0.0 │
│ track/num_transfer │ 0.0 │
│ track/num_unique_objects │ 41.0 │
│ track/partially_tracked │ 0.0 │
│ track/precision │ 31.578947067260742 │
│ track/recall │ 0.6302521228790283 │
└───────────────────────────┴───────────────────────────┘

[ReefAlturki]

Hi,
Thank you for opening the issue. I was wondering if you were able to reproduce the results?
Cheers!

[RashoAli]

Hi,
Unfortunately no

[ReefAlturki]

I am getting exactly the same as you results when testing the model weights uploaded by the developers, if you could please let me know when you solve the issue

when you pre-train the model and test it, did you face an issue like this:
dt_dets = dt[np.logical_and(dt[:, 0] == seq, dt[:, 1] == frame)][:, (2, 8, 9)]
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

Thank you!

[RashoAli]

i stopped testing with the model :)
unfortunately i cant help you there