Large performance gap between trained model using default setting and the provided trained model.

[bityangke]

With the provided trained 'resnet38_SEAM.pth', the results of SEAM step evaluation:

0/60 background score: 0.000 mIoU: 28.861%
1/60 background score: 0.010 mIoU: 32.021%
2/60 background score: 0.020 mIoU: 35.937%
3/60 background score: 0.030 mIoU: 39.372%
4/60 background score: 0.040 mIoU: 42.470%
5/60 background score: 0.050 mIoU: 45.309%
6/60 background score: 0.060 mIoU: 47.967%
7/60 background score: 0.070 mIoU: 50.436%
8/60 background score: 0.080 mIoU: 52.721%
9/60 background score: 0.090 mIoU: 54.865%
10/60 background score: 0.100 mIoU: 56.885%
11/60 background score: 0.110 mIoU: 58.777%
12/60 background score: 0.120 mIoU: 60.595%
13/60 background score: 0.130 mIoU: 62.310%
14/60 background score: 0.140 mIoU: 63.905%
15/60 background score: 0.150 mIoU: 65.372%
16/60 background score: 0.160 mIoU: 66.710%
17/60 background score: 0.170 mIoU: 67.907%
18/60 background score: 0.180 mIoU: 68.925%
19/60 background score: 0.190 mIoU: 69.758%
20/60 background score: 0.200 mIoU: 70.414%
21/60 background score: 0.210 mIoU: 71.014%
22/60 background score: 0.220 mIoU: 71.291%
23/60 background score: 0.230 mIoU: 71.324%
24/60 background score: 0.240 mIoU: 71.143%
25/60 background score: 0.250 mIoU: 70.799%
26/60 background score: 0.260 mIoU: 70.287%
27/60 background score: 0.270 mIoU: 69.664%
28/60 background score: 0.280 mIoU: 68.952%
29/60 background score: 0.290 mIoU: 68.148%
30/60 background score: 0.300 mIoU: 67.274%
31/60 background score: 0.310 mIoU: 66.322%
32/60 background score: 0.320 mIoU: 65.305%
33/60 background score: 0.330 mIoU: 64.232%
34/60 background score: 0.340 mIoU: 63.105%
35/60 background score: 0.350 mIoU: 61.939%
36/60 background score: 0.360 mIoU: 60.727%
37/60 background score: 0.370 mIoU: 59.485%
38/60 background score: 0.380 mIoU: 58.215%
39/60 background score: 0.390 mIoU: 56.921%
40/60 background score: 0.400 mIoU: 55.609%
41/60 background score: 0.410 mIoU: 54.281%
42/60 background score: 0.420 mIoU: 52.940%
43/60 background score: 0.430 mIoU: 51.605%
44/60 background score: 0.440 mIoU: 50.279%
45/60 background score: 0.450 mIoU: 48.955%
46/60 background score: 0.460 mIoU: 47.630%
47/60 background score: 0.470 mIoU: 46.303%
48/60 background score: 0.480 mIoU: 44.982%
49/60 background score: 0.490 mIoU: 43.653%
50/60 background score: 0.500 mIoU: 42.330%
51/60 background score: 0.510 mIoU: 41.015%
52/60 background score: 0.520 mIoU: 39.709%
53/60 background score: 0.530 mIoU: 38.409%
54/60 background score: 0.540 mIoU: 37.119%
55/60 background score: 0.550 mIoU: 35.848%
56/60 background score: 0.560 mIoU: 34.601%
57/60 background score: 0.570 mIoU: 33.372%
58/60 background score: 0.580 mIoU: 32.158%
59/60 background score: 0.590 mIoU: 30.959%

When using the 'resnet38_SEAM.pth' trained myself using the default settings (except that I used two GPU cards，the batch size was still set to 8), the results of SEAM step evaluation:

0/60 background score: 0.000 mIoU: 22.938%
1/60 background score: 0.010 mIoU: 26.294%
2/60 background score: 0.020 mIoU: 30.367%
3/60 background score: 0.030 mIoU: 33.779%
4/60 background score: 0.040 mIoU: 36.815%
5/60 background score: 0.050 mIoU: 39.461%
6/60 background score: 0.060 mIoU: 41.722%
7/60 background score: 0.070 mIoU: 43.691%
8/60 background score: 0.080 mIoU: 45.386%
9/60 background score: 0.090 mIoU: 46.875%
10/60 background score: 0.100 mIoU: 48.230%
11/60 background score: 0.110 mIoU: 49.466%
12/60 background score: 0.120 mIoU: 50.592%
13/60 background score: 0.130 mIoU: 51.575%
14/60 background score: 0.140 mIoU: 52.443%
15/60 background score: 0.150 mIoU: 53.182%
16/60 background score: 0.160 mIoU: 53.806%
17/60 background score: 0.170 mIoU: 54.334%
18/60 background score: 0.180 mIoU: 54.759%
19/60 background score: 0.190 mIoU: 55.087%
20/60 background score: 0.200 mIoU: 55.339%
21/60 background score: 0.210 mIoU: 55.510%
22/60 background score: 0.220 mIoU: 55.590%
23/60 background score: 0.230 mIoU: 55.594%
24/60 background score: 0.240 mIoU: 55.525%
25/60 background score: 0.250 mIoU: 55.382%
26/60 background score: 0.260 mIoU: 55.169%
27/60 background score: 0.270 mIoU: 54.892%
28/60 background score: 0.280 mIoU: 54.556%
29/60 background score: 0.290 mIoU: 54.155%
30/60 background score: 0.300 mIoU: 53.685%
31/60 background score: 0.310 mIoU: 53.182%
32/60 background score: 0.320 mIoU: 52.640%
33/60 background score: 0.330 mIoU: 52.064%
34/60 background score: 0.340 mIoU: 51.445%
35/60 background score: 0.350 mIoU: 50.793%
36/60 background score: 0.360 mIoU: 50.107%
37/60 background score: 0.370 mIoU: 49.380%
38/60 background score: 0.380 mIoU: 48.624%
39/60 background score: 0.390 mIoU: 47.837%
40/60 background score: 0.400 mIoU: 47.029%
41/60 background score: 0.410 mIoU: 46.199%
42/60 background score: 0.420 mIoU: 45.353%
43/60 background score: 0.430 mIoU: 44.483%
44/60 background score: 0.440 mIoU: 43.593%
45/60 background score: 0.450 mIoU: 42.681%
46/60 background score: 0.460 mIoU: 41.749%
47/60 background score: 0.470 mIoU: 40.809%
48/60 background score: 0.480 mIoU: 39.855%
49/60 background score: 0.490 mIoU: 38.890%
50/60 background score: 0.500 mIoU: 37.914%
51/60 background score: 0.510 mIoU: 36.934%
52/60 background score: 0.520 mIoU: 35.954%
53/60 background score: 0.530 mIoU: 34.974%
54/60 background score: 0.540 mIoU: 33.988%
55/60 background score: 0.550 mIoU: 32.998%
56/60 background score: 0.560 mIoU: 32.011%
57/60 background score: 0.570 mIoU: 31.033%
58/60 background score: 0.580 mIoU: 30.064%
59/60 background score: 0.590 mIoU: 29.102%

[bityangke]

Did anyone encounter the same situation or have ideas about the situation?

[halbielee]

which dataset (among train/train_aug/val) do you use? is it train? and npy or png?

I think the performance of your custom trained version is similar to that of my execution (pretrained / custom both)

The performance of your pretrained implementation is too high.

[bityangke]

@halbielee Hi, I use the default training setting of this repo, that is, using voc12/train_aug.txt to train the model and evaluating on VOC2012/ImageSets/Segmentation/train.txt.
The pretrained implementation I used is the trained model provided by the author @YudeWang.

[halbielee]

@bityangke
I also follow the same setting and I get different result of yours.
Hmm, which pretrained model do you use on Google Drive? or Baidu?
I used the pretrained model on Google Drive and I got similar result of your custom learning.

Can you share the exact pretrained weight file [resnet38_SEAM.pth]?
I will execute with it and let you know the result.

[bityangke]

@halbielee Hi I use the pre-trained model on Google Drive

[halbielee]

@bityangke
I got the result from the pretrained model!

0/60 background score: 0.000 mIoU: 23.150%
1/60 background score: 0.010 mIoU: 25.909%
2/60 background score: 0.020 mIoU: 29.300%
3/60 background score: 0.030 mIoU: 32.255%
4/60 background score: 0.040 mIoU: 34.881%
5/60 background score: 0.050 mIoU: 37.225%
6/60 background score: 0.060 mIoU: 39.358%
7/60 background score: 0.070 mIoU: 41.260%
8/60 background score: 0.080 mIoU: 42.931%
9/60 background score: 0.090 mIoU: 44.451%
10/60 background score: 0.100 mIoU: 45.833%
11/60 background score: 0.110 mIoU: 47.077%
12/60 background score: 0.120 mIoU: 48.244%
13/60 background score: 0.130 mIoU: 49.304%
14/60 background score: 0.140 mIoU: 50.253%
15/60 background score: 0.150 mIoU: 51.123%
16/60 background score: 0.160 mIoU: 51.901%
17/60 background score: 0.170 mIoU: 52.578%
18/60 background score: 0.180 mIoU: 53.139%
19/60 background score: 0.190 mIoU: 53.637%
20/60 background score: 0.200 mIoU: 54.066%
21/60 background score: 0.210 mIoU: 54.565%
22/60 background score: 0.220 mIoU: 54.890%
23/60 background score: 0.230 mIoU: 55.141%
24/60 background score: 0.240 mIoU: 55.310%
25/60 background score: 0.250 mIoU: 55.391%
26/60 background score: 0.260 mIoU: 55.406%
27/60 background score: 0.270 mIoU: 55.346%
28/60 background score: 0.280 mIoU: 55.218%
29/60 background score: 0.290 mIoU: 55.034%
30/60 background score: 0.300 mIoU: 54.785%
31/60 background score: 0.310 mIoU: 54.476%
32/60 background score: 0.320 mIoU: 54.109%
33/60 background score: 0.330 mIoU: 53.684%
34/60 background score: 0.340 mIoU: 53.216%
35/60 background score: 0.350 mIoU: 52.717%
36/60 background score: 0.360 mIoU: 52.175%
37/60 background score: 0.370 mIoU: 51.596%
38/60 background score: 0.380 mIoU: 50.993%
39/60 background score: 0.390 mIoU: 50.355%
40/60 background score: 0.400 mIoU: 49.673%
41/60 background score: 0.410 mIoU: 48.949%
42/60 background score: 0.420 mIoU: 48.186%
43/60 background score: 0.430 mIoU: 47.392%
44/60 background score: 0.440 mIoU: 46.578%
45/60 background score: 0.450 mIoU: 45.736%
46/60 background score: 0.460 mIoU: 44.870%
47/60 background score: 0.470 mIoU: 43.983%
48/60 background score: 0.480 mIoU: 43.076%
49/60 background score: 0.490 mIoU: 42.141%
50/60 background score: 0.500 mIoU: 41.182%
51/60 background score: 0.510 mIoU: 40.207%
52/60 background score: 0.520 mIoU: 39.217%
53/60 background score: 0.530 mIoU: 38.211%
54/60 background score: 0.540 mIoU: 37.193%
55/60 background score: 0.550 mIoU: 36.172%
56/60 background score: 0.560 mIoU: 35.153%
57/60 background score: 0.570 mIoU: 34.131%
58/60 background score: 0.580 mIoU: 33.108%
59/60 background score: 0.590 mIoU: 32.082%

I got the same result and the number 55.406 % is the number on the paper Table 1.

[bityangke]

@halbielee My bad. I need the CAM generated by PSA for a downstream task, so I put the PSA CAM in VOC2012/SegmentationClass/ to replace the original GT images. I evaluated the custom model on another machine with original GT data, so the result was not affected; when I changed to the correct GT, the result of pretrained model was exactly the same as your result.

[hchoi71]

is the number on the paper Ta

Is pretrained model that you used referring to 'ilsvrc-cls_rna-a1_cls1000_ep-0001.params' or 'resnet38_SEAM.pth'?