> Can't reproduce the result?

[xumh-9]

Hello,
In fact, the performance is improved by hyper-parameter tuning without any model change.
Specifically, alpha: 0.0002 -> 0.0005, r_act: 8 -> 9, r_bkg: 6 -> 4
You can also find it in the options.py.
In addition, I updated the best model file, with which you can see the improved result.
Thanks!

Thanks for your reply.I test the best model that you have updated. I have changed the parameter as you said.But I can't reproduce the result.The result that I test is as follows:
Step: 0
Test_acc: 0.8905
average_mAP: 0.4038
mAP@0.1: 0.6551
mAP@0.2: 0.5836
mAP@0.3: 0.5052
mAP@0.4: 0.4158
mAP@0.5: 0.3245
mAP@0.6: 0.2266
mAP@0.7: 0.1161
Do you know the reason?

Originally posted by @xumh-9 in #9 (comment)

[Pilhyeon]

The performance may vary depending on the environment, seed, and others.
I recommend using the pre-trained model for precise reproduction.
Thanks!

[xumh-9]

The performance may vary depending on the environment, seed, and others.
I recommend using the pre-trained model for precise reproduction.
Thanks!

Thank for your reply.I used the pre-trained model that you have updated to test,but the result is :

Step: 0
Test_acc: 0.8905
average_mAP: 0.4038
mAP@0.1: 0.6551
mAP@0.2: 0.5836
mAP@0.3: 0.5052
mAP@0.4: 0.4158
mAP@0.5: 0.3245
mAP@0.6: 0.2266
mAP@0.7: 0.1161

The result is worse than your paper.I think the envirment is not detemined with the test result.And the enviroment that I test your pre-trained model is in Google Colab.Is the pre-trained model you offerd newest?

[Pilhyeon]

The environment, in fact, affects the performance.
For instance, some numpy functions behave differently according to the numpy version.
Please ensure that your environment is identical to the requirements.
For the model file, I updated it last day.

[xumh-9]

The environment, in fact, affects the performance.
For instance, some numpy functions behave differently according to the numpy version.
Please ensure that your environment is identical to the requirements.
For the model file, I updated it last day.

Thank you for your quick reply.I will make the environment same to your requirements and test the model again.

[xumh-9]

The environment, in fact, affects the performance.
For instance, some numpy functions behave differently according to the numpy version.
Please ensure that your environment is identical to the requirements.
For the model file, I updated it last day.

Hello.
Thank you for your reply. I make the environment match your requirements,but I still can't reproduce the result .And the rsult is still as before:
Step: 0
Test_acc: 0.8905
average_mAP: 0.4038
mAP@0.1: 0.6551
mAP@0.2: 0.5836
mAP@0.3: 0.5052
mAP@0.4: 0.4158
mAP@0.5: 0.3245
mAP@0.6: 0.2266
mAP@0.7: 0.1161

And my environment is:

Name                    Version                   Build  Channel |   |   |   |   |  

_libgcc_mutex             0.1                        main |   |   |   |   |  
absl-py                   0.11.0                   pypi_0    pypi |   |   |   |   |  
astor                     0.8.1                    pypi_0    pypi |   |   |   |   |  
ca-certificates           2020.12.8            h06a4308_0 |   |   |   |   |  
cached-property           1.5.2                    pypi_0    pypi |   |   |   |   |  
certifi                   2020.12.5        py36h06a4308_0 |   |   |   |   |  
docopt                    0.6.2                    pypi_0    pypi |   |   |   |   |  
future                    0.18.2                   pypi_0    pypi |   |   |   |   |  
gast                      0.2.2                    pypi_0    pypi |   |   |   |   |  
google-pasta              0.2.0                    pypi_0    pypi |   |   |   |   |  
grpcio                    1.34.0                   pypi_0    pypi |   |   |   |   |  
h5py                      3.1.0                    pypi_0    pypi |   |   |   |   |  
importlib-metadata        3.3.0                    pypi_0    pypi |   |   |   |   |  
joblib                    0.13.0                   pypi_0    pypi |   |   |   |   |  
keras-applications        1.0.8                    pypi_0    pypi |   |   |   |   |  
keras-preprocessing       1.1.2                    pypi_0    pypi |   |   |   |   |  
ld_impl_linux-64          2.33.1               h53a641e_7 |   |   |   |   |  
libedit                   3.1.20191231         h14c3975_1 |   |   |   |   |  
libffi                    3.3                  he6710b0_2 |   |   |   |   |  
libgcc-ng                 9.1.0                hdf63c60_0 |   |   |   |   |  
libstdcxx-ng              9.1.0                hdf63c60_0 |   |   |   |   |  
markdown                  3.3.3                    pypi_0    pypi |   |   |   |   |  
ncurses                   6.2                  he6710b0_1 |   |   |   |   |  
numpy                     1.19.0                   pypi_0    pypi |   |   |   |   |  
openssl                   1.1.1i               h27cfd23_0 |   |   |   |   |  
opt-einsum                3.3.0                    pypi_0    pypi |   |   |   |   |  
pandas                    0.23.4                   pypi_0    pypi |   |   |   |   |  
pillow                    8.0.1                    pypi_0    pypi |   |   |   |   |  
pip                       20.3.3           py36h06a4308_0 |   |   |   |   |  
pqi                       2.0.6                    pypi_0    pypi |   |   |   |   |  
protobuf                  3.14.0                   pypi_0    pypi |   |   |   |   |  
python                    3.6.12               hcff3b4d_2 |   |   |   |   |  
python-dateutil           2.8.1                    pypi_0    pypi |   |   |   |   |  
pytz                      2020.4                   pypi_0    pypi |   |   |   |   |  
readline                  8.0                  h7b6447c_0 |   |   |   |   |  
scikit-learn              0.20.0                   pypi_0    pypi |   |   |   |   |  
scipy                     1.1.0                    pypi_0    pypi |   |   |   |   |  
setuptools                51.0.0           py36h06a4308_2 |   |   |   |   |  
six                       1.15.0                   pypi_0    pypi |   |   |   |   |  
sqlite                    3.33.0               h62c20be_0 |   |   |   |   |  
tensorboard               1.15.0                   pypi_0    pypi |   |   |   |   |  
tensorboard-logger        0.1.0                    pypi_0    pypi |   |   |   |   |  
tensorflow                1.15.2                   pypi_0    pypi |   |   |   |   |  
tensorflow-estimator      1.15.1                   pypi_0    pypi |   |   |   |   |  
termcolor                 1.1.0                    pypi_0    pypi |   |   |   |   |  
tk                        8.6.10               hbc83047_0 |   |   |   |   |  
torch                     1.6.0                    pypi_0    pypi |   |   |   |   |  
torchvision               0.7.0                    pypi_0    pypi |   |   |   |   |  
tqdm                      4.31.1                   pypi_0    pypi |   |   |   |   |  
typing-extensions         3.7.4.3                  pypi_0    pypi |   |   |   |   |  
werkzeug                  1.0.1                    pypi_0    pypi |   |   |   |   |  
wheel                     0.36.2             pyhd3eb1b0_0 |   |   |   |   |  
wrapt                     1.12.1                   pypi_0    pypi |   |   |   |   |  
xz                        5.2.5                h7b6447c_0 |   |   |   |   |  

And my cuda versionis 10.2:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

I don't know the reason that casue the problem.Maybe you change something that you ignored,but you forgot. I hope you can help me solve this problem.Because it' s very important for me.I'm waiting for your reply.
Thank you.

[xumh-9]

The environment, in fact, affects the performance.
For instance, some numpy functions behave differently according to the numpy version.
Please ensure that your environment is identical to the requirements.
For the model file, I updated it last day.

Hello.Can you tell me why the average result is 1.5 lower than your paper?Because it's very important for me.

[Pilhyeon]

Sorry, but I have no idea about what causes the performance difference, as I cannot look into the exact status of your environment.
I tried reproducing this repo on another environment and got the same result as mine.
A possible source is that you may be using the old code.
Make sure to use the latest code as well as the model file.

[xumh-9]

Sorry, but I have no idea about what causes the performance difference, as I cannot look into the exact status of your environment.
I tried reproducing this repo on another environment and got the same result as mine.
A possible source is that you may be using the old code.
Make sure to use the latest code as well as the model file.

Thank you for your reply.I have updated the newest code and the newest model.But the results don't change.Maybe you use some tricks that you ignored。

[Pilhyeon]

I'm sorry.
I found that the link to the pre-trained model was wrong, which is corrected now.
Please update the model file and re-evaluate the model.
Thanks!

[xumh-9]

I'm sorry.
I found that the link to the pre-trained model was wrong, which is corrected now.
Please update the model file and re-evaluate the model.
Thanks!

Thank you for your reply.
I can reproduce the result that used the newest pre-trained model now.Could you please update the code again? Maybe you forgot some details.
How long can you get the best model?

[liming-ai]

I'm sorry.
I found that the link to the pre-trained model was wrong, which is corrected now.
Please update the model file and re-evaluate the model.
Thanks!

Thank you for your reply.
I can reproduce the result that used the newest pre-trained model now.Could you please update the code again? Maybe you forgot some details.
How long can you get the best model?

Hi @xumh-9 , have you reproduced the result in paper? I tried many times, but I cannot get the result as good as pretrained model, even tried different random seeds.