Difficulties on reproducing the results

[vribeiro1]

Hi, Yuan!

I'm trying to reproduce the results you published at Adversarial Learning with Multi-Scale Loss for Skin Lesion Segmentation. The problem is that I'm not able to achieve the same results as you did (mDice = 0.867). The highest number I get is around 0.832. I'm using CUDA 8.0, with Python 2.7 and PyTorch 1.2. The hyper parameters I use are the same as you describe in the paper. I'm training for 490 epochs, using ISIC 2017 train and test sets as training and validation, respectively.

The image below has the results of 5 experiments with the code available in this repo. I really appreciate if you can give me some thoughts on what I might be missing.

Thanks a lot! Best regards!

[YuanXue1993]

Hi @vribeiro1 , thanks for your interest in our work! The code here is just an example for anyone interested in the SegAN idea to try it out. The details may not be the same as we used in the paper such as the preprocessing and the different value of adaptive loss. I didn't get time to organize and fix the code so there may even be some bugs as some other people have found, thus it just serves as an example and is not for reproducing the result.

If you are more interested in getting better performance than playing with the adversarial training idea, you can refer to our updated model we used for ISIC2018. You can find our document on ISIC2018 leaderboard under my name. For the new model we got much better performance than the model we used in the paper. We also tried ISIC2017 data with the new model, where we got over 86.7 dice with a single model while the result in the paper was from a model ensemble (the single model score was around 85.5 before ensemble if I remember correctly).

[vribeiro1]

Hi, @YuanXue1993 ! Thanks for the fast response. I'm very interested in adversarial training for semantic segmentation and I got pretty excited with your work. I could run your code almost end to end. With the versions I told, it runs pretty well. I'll see what I can do about not having the same results.
Thank you again! Best regards