Unable to duplicate the original paper's results

[ang-bas]

Hello.... I am a new graduate student interested in your work. I was trying to duplicate the results you have on your paper. I ran the main.py following your Readme file, but the resulting F1 scores seem very low. I am curious as to how I should set the parameters or arguments to have the same results with yours. Thank you.

[onlyzdd]

@ang-bas Your result seems strange. Please check the following things so that I can spot the problem and help you further:

1. Do yun run the code with the same Python package requirements? If not, try to run code with the same requirements.
2. Have you made any changes to the source code and dataset? If yes, paste your modification here.
3. Are you using exact the same command in the README (which produce the result showing in the screenshot)? If not, paste your command here.
4. Please paste logs for all epochs here.

[ang-bas]

Dear Mr. Dongdong, Zhang Thank you for your reply, we are very appreciative, We are using the same commands on the README file. We did not change any parameter settings either. We used different versions of libraries, of course. We used: Python 3.9 Matplotlib 3.7.1 Numpy 1.22.3 Pandas 2.0.1 PyTorch 1.12.1 Scikit-learn 1.2.2 Scipy 1.10.1 Shap 0.40.0 Tqdm 4.65.0 Wfdb 4.1.1 But we did not think that would influence the F1 scores of your model. Regards, Angger Bas

…

On Wed, 17 May 2023 at 08:17, Dongdong Zhang ***@***.***> wrote: @ang-bas <https://github.com/ang-bas> Your result seems strange. Please check the following things so that I can spot the problem and help you further: 1. Do yun run the code with the same Python package requirements? If not, try to run code with the same requirements. 2. Have you made any changes to the source code and dataset? If yes, paste your modification here. 3. Are you using exact the same command in the README (which produce the result showing in the screenshot)? If not, paste your command here. 4. Please paste logs for all epochs here. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXPV2BKV56NVB7WKWEJ5ZNTXGQKJPANCNFSM6AAAAAAYBW6QBY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[onlyzdd]

@ang-bas Have you closely followed the instructions in the README? I just downloaded the code fro github and data from dropbox, ran model training after preprocessing, and got the output:

Notice that I only show first 5 epochs here. But around 20-30 epochs, F1 on the evaluation dataset achieves/exceeds the reported result.
However, in your screenshot, there are some problems:

1. The training loss is not going down, which means the model is not training at all.
2. The validation loss is bigger than training loss, which seems impossible since number of validation samples is less than training (1:8).
3. All AUCs during validation are around 0.5, which also indicates the model is not learning.
   BTW, I ran the code with the following package requirements. So I would suggest you to re-download the code and data, run preprocessing, and run training with no changes.

[ang-bas]

Dear Mr. Dongdong, Zhang Thank you for your knowledge... The event happened to my study when: 1. I tried to run the code with the exact same versions of libraries you have listed and the errors show up (as shown on attached screenshot). 2. I used another conda environment to run newer versions of these. They work but the F1 scores are low (as shown on attached screenshot). best regards, angger

…

On Fri, 19 May 2023 at 09:34, Dongdong Zhang ***@***.***> wrote: @ang-bas <https://github.com/ang-bas> Have you closely followed the instructions in the README? I just downloaded the code fro github and data from dropbox <https://www.dropbox.com/s/unicm8ulxt24vh8/CPSC.zip?dl=0>, ran model training after preprocessing, and got the output: [image: image] <https://user-images.githubusercontent.com/13433549/239412083-9cacc9ca-dca1-4959-ac23-5460f89ee678.png> Notice that I only show first 5 epochs here. But around 20-30 epochs, F1 on the evaluation dataset achieves/exceeds the reported result. However, in your screenshot, there are some problems: 1. The training loss is not going down, which means the model is not training at all. 2. The validation loss is bigger than training loss, which seems impossible since number of validation samples is less than training (1:8). 3. All AUCs during validation are around 0.5, which also indicates the model is not learning. BTW, I ran the code with the following package requirements. So I would suggest you to re-download the code and data, run preprocessing, and run training with no changes. [image: image] <https://user-images.githubusercontent.com/13433549/239414034-11d59554-9019-4ceb-9c08-69066fa6af24.png> — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXPV2BLC6PL27UOPBTRRUE3XG3EYZANCNFSM6AAAAAAYBW6QBY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[onlyzdd]

Hi @ang-bas, I'm sorry to hear that the code is not working for you and I'm unable to locate the problem according to your description.
To help you, I created a Colab Notebook which works well. I would also suggest you to run the code with the data on another machine to check whether it works or not.

[onlyzdd]

@ang-bas I've also ran the code in the environment with the same package versions you are using. The result agrees with it as reported in the paper.
It's been a long time since your last feedback. Have you tried my suggestions?
As I've attached the Colab Notebook before which is working well. And there're some works based on this project, ECG analysis Using Deep Learning, which shows the results are successfully reproduced.
So now I'm closing this issue as I cannot reproduce your problem. But feel free to reopen and comment with your settings and results.