Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only got 76.90% over Pascal VOC2012 val set #41

Open
buttomnutstoast opened this issue Nov 21, 2017 · 5 comments
Open

Only got 76.90% over Pascal VOC2012 val set #41

buttomnutstoast opened this issue Nov 21, 2017 · 5 comments

Comments

@buttomnutstoast
Copy link

buttomnutstoast commented Nov 21, 2017 •
edited

Hi,

I tried the Pascal VOC2012 trained model provided in the repository. However, I only got 76.90%, instead of 80.84% reported in the README.. I used the latest MXNet (v0.11.0). Do you have any idea?

Here is my log for the last three images.

2017-11-20 16:25:04,536 Host Done 1447/1449 with speed: 1.64/s
2017-11-20 16:25:04,537 Host pixel acc: 94.11%, mean acc: 82.53%, mean iou: 76.05%
2017-11-20 16:25:04,537 Host
[97.46 90.25 55.71 89.61 72.90 84.55 95.42 90.71 94.92 54.97 86.99 61.03
90.37 90.42 89.59 92.72 69.21 88.79 61.88 89.36 86.20]
2017-11-20 16:25:04,537 Host
[93.12 85.17 53.60 86.79 65.93 75.40 92.33 85.52 90.12 44.63 81.11 53.32
84.97 81.70 82.75 85.10 63.58 85.11 47.66 82.99 76.17]
2017-11-20 16:25:05,090 Host Done 1448/1449 with speed: 1.64/s
2017-11-20 16:25:05,091 Host pixel acc: 94.11%, mean acc: 82.53%, mean iou: 76.06%
2017-11-20 16:25:05,091 Host
[97.46 90.25 55.71 89.61 72.90 84.55 95.42 90.71 94.92 54.97 86.99 61.03
90.39 90.42 89.59 92.72 69.21 88.79 61.95 89.36 86.20]
2017-11-20 16:25:05,091 Host
[93.11 85.17 53.60 86.79 65.93 75.40 92.33 85.52 90.12 44.63 81.11 53.32
84.98 81.70 82.75 85.10 63.58 85.11 47.88 82.99 76.17]
2017-11-20 16:25:05,788 Host Done 1449/1449 with speed: 1.64/s
2017-11-20 16:25:05,788 Host pixel acc: 94.11%, mean acc: 82.53%, mean iou: 76.09%
2017-11-20 16:25:05,789 Host
[97.46 90.25 55.71 89.61 72.90 84.62 95.42 90.71 94.92 54.97 86.99 61.03
90.39 90.42 89.59 92.72 69.21 88.79 61.95 89.36 86.20]
2017-11-20 16:25:05,789 Host
[93.10 85.17 53.60 86.79 65.93 76.00 92.33 85.52 90.12 44.63 81.11 53.32
84.98 81.70 82.75 85.10 63.58 85.11 47.88 82.99 76.17]
2017-11-20 16:25:05,789 Host Done in 882.66 s.
@mylxiaoyi
Copy link

I simply ported to python3 and tried with mxnet 1.0, then get the results as the author's.

2017-12-07 21:43:22,509 Host Done 1449/1449 with speed: 0.18/s
2017-12-07 21:43:22,509 Host pixel acc: 95.70%, mean acc: 86.26%, mean iou: 80.84%
2017-12-07 21:43:22,509 Host
[98.54 95.84 86.65 93.99 85.36 87.12 95.66 92.31 97.00 54.13 90.22 60.82
92.64 92.83 93.37 93.17 76.37 91.94 62.55 88.68 82.28]
2017-12-07 21:43:22,510 Host
[95.17 92.26 69.49 91.99 77.32 80.42 93.83 89.02 92.56 44.12 86.38 57.93
88.94 86.07 87.06 88.22 70.05 89.14 53.59 85.80 78.37]
2017-12-07 21:43:22,510 Host Done in 8155.81 s.

2017-12-08 00:33:29,299 Host Done 1449/1449 with speed: 0.19/s
2017-12-08 00:33:29,299 Host pixel acc: 96.22%, mean acc: 87.79%, mean iou: 82.87%
2017-12-08 00:33:29,299 Host
[98.52 96.00 87.58 93.95 82.92 81.21 97.88 92.68 97.04 59.27 95.60 69.96
92.80 95.10 93.32 94.92 73.03 95.16 72.35 92.74 81.57]
2017-12-08 00:33:29,300 Host
[95.66 93.92 70.93 90.37 77.40 75.07 96.51 90.65 92.08 49.72 93.36 65.84
88.20 91.67 89.10 90.44 68.30 92.45 59.21 90.17 79.18]
2017-12-08 00:33:29,300 Host Done in 7752.58 s.

maybe you would try the newest version, mxnet 1.0

@mcever
Copy link

mcever commented Mar 6, 2019 •
edited

I too am now trying to reproduce results. It would be really helpful if we could get version specifications for packages used to achieve the published results (e.g. a pip requirements.txt would be great). I am achieving 75-76% using:

(ademxapp) austin@vader:~/ssd/ademxapp$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
certifi==2018.11.29
chardet==3.0.4
graphviz==0.8.1
h5py==2.9.0
idna==2.6
mxnet-cu90==1.4.0
numpy==1.13.3
Pillow==5.4.1
PyYAML==3.13
requests==2.21.0
six==1.12.0
urllib3==1.22

(ademxapp) austin@vader:~/ssd/ademxapp$ python issegm/voc.py --data-root data/VOCdevkit --output output --phase val --weights models/voc_rna-a1_cls21_s8_coco_ep-0001.params --split val --test-scales 500 --test-flipping --gpus 0
. . .
2019-03-05 16:06:11,561 Host pixel acc: 93.92%, mean acc: 87.64%, mean iou: 75.17%
2019-03-05 16:06:11,562 Host 
[95.27 96.00 87.58 93.95 82.92 81.21 97.88 92.68 97.04 59.27 95.60 69.96
 92.80 95.10 93.32 94.92 73.03 95.16 72.35 92.74 81.57]
2019-03-05 16:06:11,562 Host 
[92.70 83.09 44.98 83.29 67.80 70.50 92.56 85.14 86.84 43.02 84.93 61.79
 83.36 83.14 78.65 82.37 56.09 83.02 55.05 85.35 74.76]
2019-03-05 16:06:11,562 Host Done in 815.40 s.
. . .
. . .
(ademxapp) austin@vader:~/ssd/ademxapp$ python issegm/voc.py --data-root data/VOCdevkit --output output --phase val --weights models/voc_rna-a1_cls21_s8_ep-0001.params --split val --test-scales 500 --te
st-flipping --gpus 0
2019-03-08 14:17:06,440 Host pixel acc: 93.51%, mean acc: 86.12%, mean iou: 73.47%
2019-03-08 14:17:06,441 Host
[95.39 95.84 86.65 93.99 85.36 87.25 95.66 92.31 97.00 54.13 90.22 60.82 
 92.64 92.83 93.37 93.17 76.37 91.94 62.55 88.68 82.28]
2019-03-08 14:17:06,441 Host
[92.37 81.35 44.01 84.87 66.81 74.85 90.05 83.48 87.34 39.09 79.47 56.18 
 83.87 78.38 77.38 80.51 57.96 80.32 50.04 81.35 73.16]
2019-03-08 14:17:07,063 Host Done 1449/1449 with speed: 1.82/s
2019-03-08 14:17:07,063 Host pixel acc: 93.50%, mean acc: 86.11%, mean iou: 73.50%
2019-03-08 14:17:07,064 Host
[95.40 95.84 86.65 93.99 85.36 87.12 95.66 92.31 97.00 54.13 90.22 60.82 
 92.64 92.83 93.37 93.17 76.37 91.94 62.55 88.68 82.28]
2019-03-08 14:17:07,064 Host
[92.36 81.35 44.01 84.87 66.81 75.55 90.05 83.48 87.34 39.09 79.47 56.18 
 83.87 78.38 77.38 80.51 57.96 80.32 50.04 81.35 73.16]
2019-03-08 14:17:07,064 Host Done in 798.10 s.

I have begun trying to port the code to Python 3, but I am hoping to retrain this network on my own data, so I fear that might be complicated, but I'll plan to post an update later.

@buttomnutstoast
Copy link
Author

buttomnutstoast commented Mar 6, 2019 via email

@mcever
Copy link

mcever commented Mar 6, 2019 •
edited

I have not begun training yet. I was hoping I could simply reproduce the reported results with voc_rna-a1_cls21_s8_coco_ep-0001.params before trying to train on my own data. You can see exactly the command and its output above.

@czzerone
Copy link

I too am now trying to reproduce results. It would be really helpful if we could get version specifications for packages used to achieve the published results (e.g. a pip requirements.txt would be great). I am achieving 75-76% using:

(ademxapp) austin@vader:~/ssd/ademxapp$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
certifi==2018.11.29
chardet==3.0.4
graphviz==0.8.1
h5py==2.9.0
idna==2.6
mxnet-cu90==1.4.0
numpy==1.13.3
Pillow==5.4.1
PyYAML==3.13
requests==2.21.0
six==1.12.0
urllib3==1.22

(ademxapp) austin@vader:~/ssd/ademxapp$ python issegm/voc.py --data-root data/VOCdevkit --output output --phase val --weights models/voc_rna-a1_cls21_s8_coco_ep-0001.params --split val --test-scales 500 --test-flipping --gpus 0
. . .
2019-03-05 16:06:11,561 Host pixel acc: 93.92%, mean acc: 87.64%, mean iou: 75.17%
2019-03-05 16:06:11,562 Host 
[95.27 96.00 87.58 93.95 82.92 81.21 97.88 92.68 97.04 59.27 95.60 69.96
 92.80 95.10 93.32 94.92 73.03 95.16 72.35 92.74 81.57]
2019-03-05 16:06:11,562 Host 
[92.70 83.09 44.98 83.29 67.80 70.50 92.56 85.14 86.84 43.02 84.93 61.79
 83.36 83.14 78.65 82.37 56.09 83.02 55.05 85.35 74.76]
2019-03-05 16:06:11,562 Host Done in 815.40 s.
. . .
. . .
(ademxapp) austin@vader:~/ssd/ademxapp$ python issegm/voc.py --data-root data/VOCdevkit --output output --phase val --weights models/voc_rna-a1_cls21_s8_ep-0001.params --split val --test-scales 500 --te
st-flipping --gpus 0
2019-03-08 14:17:06,440 Host pixel acc: 93.51%, mean acc: 86.12%, mean iou: 73.47%
2019-03-08 14:17:06,441 Host
[95.39 95.84 86.65 93.99 85.36 87.25 95.66 92.31 97.00 54.13 90.22 60.82 
 92.64 92.83 93.37 93.17 76.37 91.94 62.55 88.68 82.28]
2019-03-08 14:17:06,441 Host
[92.37 81.35 44.01 84.87 66.81 74.85 90.05 83.48 87.34 39.09 79.47 56.18 
 83.87 78.38 77.38 80.51 57.96 80.32 50.04 81.35 73.16]
2019-03-08 14:17:07,063 Host Done 1449/1449 with speed: 1.82/s
2019-03-08 14:17:07,063 Host pixel acc: 93.50%, mean acc: 86.11%, mean iou: 73.50%
2019-03-08 14:17:07,064 Host
[95.40 95.84 86.65 93.99 85.36 87.12 95.66 92.31 97.00 54.13 90.22 60.82 
 92.64 92.83 93.37 93.17 76.37 91.94 62.55 88.68 82.28]
2019-03-08 14:17:07,064 Host
[92.36 81.35 44.01 84.87 66.81 75.55 90.05 83.48 87.34 39.09 79.47 56.18 
 83.87 78.38 77.38 80.51 57.96 80.32 50.04 81.35 73.16]
2019-03-08 14:17:07,064 Host Done in 798.10 s.

I have begun trying to port the code to Python 3, but I am hoping to retrain this network on my own data, so I fear that might be complicated, but I'll plan to post an update later.

@mcever hello，I follow your setting for python2.7, and I test the model with voc_rna-a1_cls21_s8_coco_ep-0001.params provided by the author. I only got the miou result is 45.38%. Do you know the reason or can you give me some advice? Thanks~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mylxiaoyi @buttomnutstoast @mcever @czzerone