Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Yolov5DeepSORTwithOSNet vs Yolov5StrongSORTwithOSNet ablation study on MOT16 #33

Closed
mikel-brostrom opened this issue Jun 8, 2022 · 5 comments
Closed

Yolov5DeepSORTwithOSNet vs Yolov5StrongSORTwithOSNet ablation study on MOT16 #33

mikel-brostrom opened this issue Jun 8, 2022 · 5 comments
Labels
Good Contributions

Comments

@mikel-brostrom
Copy link
Contributor

mikel-brostrom commented Jun 8, 2022 •
edited

Just though somebody could find this interesting. I use a modest Yolov5m as object detector:

Yolov5DeepSORTwithOSNet

HOTA: rep_1280-pedestrian          HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           51.286    50.839    52.219    55.475    75.31     56.777    78.178    81.874    53.742    66.59     76.683    51.063    
CLEAR: rep_1280-pedestrian         MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag           
COMBINED                           59.955    79.281    60.364    67.013    90.974    32.495    47.002    20.503    46.07     73987     36420     7341      452       168       243       106       2243      
Identity: rep_1280-pedestrian      IDF1      IDR       IDP       IDTP      IDFN      IDFP       
COMBINED                           64.31     55.841    75.807    61652     48755     19676

Yolov5StrongSORTwithOSNet (ecc)

HOTA: rep_1280_ecc-pedestrian      HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           51.466    50.976    52.455    55.616    75.439    56.993    78.435    81.965    53.928    66.705    76.815    51.239    

CLEAR: rep_1280_ecc-pedestrian     MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag         
COMBINED                           60.054    79.382    60.462    67.093    91.006    32.689    47.195    20.116    46.221    74075     36332     7321      450       169       244       104       2246      

Identity: rep_1280_ecc-pedestrian  IDF1      IDR       IDP       IDTP      IDFN      IDFP      
COMBINED                           64.399    55.938    75.876    61760     48647     19636

Yolov5StrongSORTwithOSNet (ecc + woc)

HOTA: rep_1280_ecc_woc-pedestrian  HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           51.458    50.914    52.476    55.591    75.301    57.536    77.498    81.906    53.934    66.897    76.706    51.314    

CLEAR: rep_1280_ecc_woc-pedestrian MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag         
COMBINED                           60.069    79.378    60.532    67.179    90.997    32.108    47.582    20.309    46.215    74170     36237     7338      512       166       246       105       2288      

Identity: rep_1280_ecc_woc-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP        
COMBINED                           64.942    56.443    76.455    62317     48090     19191

Yolov5StrongSORTwithOSNet (ecc + woc + nsa)

HOTA: rep_1280_ecc_woc_nsa-pedestrianHOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           51.521    50.654    52.851    55.268    74.918    57.776    77.45     81.545    53.98     67.309    76.301    51.357    

CLEAR: rep_1280_ecc_woc_nsa-pedestrianMOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag            
COMBINED                           59.975    78.905    60.416    67.094    90.948    30.948    48.936    20.116    45.822    74076     36331     7373      486       160       253       104       2190      

Identity: rep_1280_ecc_woc_nsa-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP         
COMBINED                           65.139    56.597    76.719    62487     47920     18962

Yolov5StrongSORTwithOSNet (ecc + woc + nsa + ema)

HOTA: rep_1280_ecc_woc_nsa_ema-pedestrianHOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           52.897    51.039    55.381    55.638    75.538    60.179    79.446    81.967    55.407    68.425    76.875    52.602    

CLEAR: rep_1280_ecc_woc_nsa_ema-pedestrianMOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag          
COMBINED                           60.217    79.392    60.696    67.176    91.203    31.915    47.969    20.116    46.373    74167     36240     7154      529       165       248       104       2185      

Identity: rep_1280_ecc_woc_nsa_ema-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP         
COMBINED                           66.877    58.068    78.837    64111     46296     17210

Yolov5StrongSORTwithOSNet (ecc + woc + nsa + ema + mc)

HOTA: rep_1280_ecc_woc_nsa_mc_ema-pedestrianHOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)

COMBINED                           52.784    51.111    55.072    55.625    75.742    59.143    81.179    82.009    55.244    67.947    77.086    52.377    

CLEAR: rep_1280_ecc_woc_nsa_mc_ema-pedestrianMOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag          
COMBINED                           60.174    79.416    60.646    67.043    91.289    31.721    47.969    20.309    46.373    74020     36387     7063      521       164       248       105       2106      

Identity: rep_1280_ecc_woc_nsa_mc_ema-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP          
COMBINED                           66.425    57.604    78.437    63599     46808     17484

So, in conclusion:

Yolov5 + StrongSORT with OSNet (with BoT (OSNet), ECC, NSA, EMA, MC, woC and no AFLink nor GSI as it is only intended to be used online)

HOTA: 52.784 (+1.498)
MOTA: 60.174 (+0.219)
IDF1: 66.425 (+2.115)

Yolov5 + DeepSORT with OSNet

HOTA: 51.286
MOTA: 59.955
IDF1: 64.31

Small boost on all major metrics 😄
@mikel-brostrom
Copy link
Contributor Author

Any idea why matching with motion cost (MC) isn't giving the boost it should?

@dyhBUPT
Copy link
Owner

dyhBUPT commented Jun 9, 2022

Thanks very much for your share of the abliation study on MOT16.

For the MC, it depends. Intuitively, tuning hyperparameters would help.

@mikel-brostrom
Copy link
Contributor Author

For the MC, it depends. Intuitively, tuning hyperparameters would help.

Thanks. Will try different values and check if I can get a boost from MC as well

@mikel-brostrom
Copy link
Contributor Author

mikel-brostrom commented Jun 9, 2022 •
edited

Got a relatively big boost from MC compared to: BoT, ECC, NSA, EMA and woC, by increasing MC_lambda to 0.995

HOTA: 995-pedestrian               HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
COMBINED                           52.948    51.068    55.46     55.639    75.608    59.992    80.202    81.989    55.447    68.353    76.972    52.613    

CLEAR: 995-pedestrian              MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag          
COMBINED                           60.265    79.402    60.705    67.147    91.246    31.915    47.776    20.309    46.434    74135     36272     7112      486       165       247       105       2164      

Identity: 995-pedestrian           IDF1      IDR       IDP       IDTP      IDFN      IDFP       
COMBINED                           66.889    58.056    78.893    64098     46309     17149

This leads to the following final results:

HOTA: 52.948 (+1.662)
MOTA: 60.265 (+0.310)
IDF1: 66.889 (+2.579)

Yolov5 + DeepSORT with OSNet

HOTA: 51.286
MOTA: 59.955
IDF1: 64.31

Thanks for making the code available and a pleasant paper read.

@dyhBUPT
Copy link
Owner

dyhBUPT commented Jun 9, 2022

Glad to see the satisfactory results and thanks again for your sharing of the ablation study.

Bese wishes.

@dyhBUPT dyhBUPT closed this as completed Jun 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Good Contributions
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikel-brostrom @dyhBUPT