reproruce acc of R3D in the paper

[CODE-SUBMIT]

hi

Great work, I rerun your checkpoint but can not reproduce your results of your paper.

My results of rerunning your checkpoint is

Overall_acc 0.899

background acc 0.984 | closet 0.659 | Bathroom 0.817 |LivingRoom&Kitchen&DiningRoom 0.819 |BedRoom 0.760 |Hall 0.641 | Balcony 0.716 | Wall 0.957 | Door 0.774

Is it a reasonable one.

[lzd1]

Hello, your result is great, but I can't get this result with their code. And when I use the balanced loss, the network can't predict the room type result.
My results is
overall accuracy 0.5211
room-type: mean accuracy 0.07095, room-type+bd: mean accuracy 0.2195
room type 0th, accuracy = 0.4967
room type 1th, accuracy = 0.0
room type 2th, accuracy = 0.0
room type 3th, accuracy = 0.0
room type 4th, accuracy = 0.0
room type 5th, accuracy = 0.0
room type 6th, accuracy = 0.0
room type 9th, accuracy = 0.7261
room type 10th, accuracy = 0.7531
.
So is the balanced loss useful ?

[andrew-begain]

@CODE-SUBMIT @lzd1 i have face the same problem with @lzd1，so , what should i do to reach the result as @CODE-SUBMIT,thanks

[lzd1]

When I removed the balanced loss, I can get the room type result which lower than the result in paper.

[andrew-begain]

When I removed the balanced loss, I can get the room type result which lower than the result in paper.

thanks for your quik reply,i change the main.py as follow image,

but when i run scores.py again,i got the same result ,

Could you please tell me where to modify it? thanks

[andrew-begain]

When I removed the balanced loss, I can get the room type result which lower than the result in paper.

thanks for your quik reply,i change the main.py as follow image,

but when i run scores.py again,i got the same result ,

Could you please tell me where to modify it? thanks

@lzd1 main.py change as follow

[lzd1]

Yes, you can print information to determine which loss to use for training

[andrew-begain]

Yes, you can print information to determine which loss to use for training

thank you very much!!!

[zhuao1997]

@lzd1 Hello, have you solved the appeal problem? I met the similar problem with you?
my result:
overall accuracy 0.607
room-type: mean accuracy 0.1401, room-type+bd: mean accuracy 0.214
room type 0th, accuracy = 0.9806
room type 1th, accuracy = 0.0
room type 2th, accuracy = 0.0
room type 3th, accuracy = 0.0
room type 4th, accuracy = 0.0
room type 5th, accuracy = 0.0
room type 6th, accuracy = 0.0
room type 9th, accuracy = 0.0
room type 10th, accuracy = 0.9451

[zhuao1997]

@andrew-begain Hello, have you solved the problem?

[lzd1]

@zhuao1997 Yes, I can get the room prediction result by not using the balanced loss as shown in the previous answer.

[zhuao1997]

@lzd1 Thank you for your reply. I run 'python main.py --pharse=Train' ,and don't use the balanced loss ,The results are the same
step 170: loss = 0.688; 5.6 data/sec, excuted 0 minutes
step 171: loss = 0.523; 5.5 data/sec, excuted 0 minutes
step 172: loss = 0.569; 5.5 data/sec, excuted 0 minutes
step 173: loss = 0.57; 5.5 data/sec, excuted 0 minutes
step 174: loss = 0.416; 5.5 data/sec, excuted 0 minutes
step 175: loss = 0.528; 5.4 data/sec, excuted 0 minutes
step 176: loss = 0.738; 5.5 data/sec, excuted 0 minutes
step 177: loss = 0.9; 5.4 data/sec, excuted 0 minutes
step 178: loss = 0.639; 5.6 data/sec, excuted 0 minutes
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> Model at epoch 0: overall accuracy = 0.607, mean_acc = 0.2157
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 0th label: accuracy = 0.9787
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 1th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 2th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 3th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 4th label: accuracy = 6.142e-07
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 5th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 6th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 9th label: accuracy = 0.0
<_io.TextIOWrapper name='EVAL_pretrained' mode='a' encoding='UTF-8'> epoch 0: 10th label: accuracy = 0.9625
step 179: loss = 0.609; 5.4 data/sec, excuted 0 minutes
step 180: loss = 0.629; 5.4 data/sec, excuted 0 minutes
step 181: loss = 0.713; 5.5 data/sec, excuted 0 minutes
step 182: loss = 0.602; 5.5 data/sec, excuted 0 minutes

I wonder if you can run out of the accuracy of each category, or have you changed the code somewhere else ?

[zhuao1997]

@lzd1 Oh, I see, after I run to 10,000 iterations, each category shows accuracy, thank you.