Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't reproduce DeepFilterNet2 and DeepFilterNet3 results on Voicebank Demand test set #537

Closed
WoodieDudy opened this issue Mar 28, 2024 · 7 comments
Closed

Can't reproduce DeepFilterNet2 and DeepFilterNet3 results on Voicebank Demand test set #537

WoodieDudy opened this issue Mar 28, 2024 · 7 comments
Labels
stale

Comments

@WoodieDudy
Copy link

WoodieDudy commented Mar 28, 2024 •
edited
Loading

I am currently on last main commit f2445da

I downloaded Voicebank Demand test set from here
I evaluated DeepFilterNet versions 1-3 using the test_voicebank_demand.py script.

Results of DeepFilterNet1 are the same as those reported in the DFN3 paper
However, the results of DFN2 and 3 and are much lower.
image

How I can reproduce results similar to those in the paper?

python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet/
2024-03-28 08:47:53 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:47:53 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:47:53 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:47:53 | INFO     | DF | Loading model settings of DeepFilterNet
2024-03-28 08:47:53 | INFO     | DF | Initializing model `deepfilternet`
2024-03-28 08:47:53 | WARNING  | DF | Could not find `best` checkpoint. Checking for default...
2024-03-28 08:47:53 | INFO     | DF | Found checkpoint models/DeepFilterNet/checkpoints/model_30.ckpt with epoch 30
2024-03-28 08:47:53 | INFO     | DF | Running on device cpu
2024-03-28 08:47:53 | INFO     | DF | Model loaded
2024-03-28 08:49:56 | INFO     | DF | Progress: 25%
2024-03-28 08:52:02 | INFO     | DF | Progress: 50%
2024-03-28 08:54:06 | INFO     | DF | Progress: 75%
2024-03-28 08:56:09 | INFO     | DF | Progress: 100%
2024-03-28 08:56:09 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:56:10 | INFO     | DF | Enhanced STOI: 0.9420673982468806
2024-03-28 08:56:10 | INFO     | DF | Enhanced PESQ: 2.809007406234741
2024-03-28 08:56:10 | INFO     | DF | Enhanced CSIG: 4.124786853790283
2024-03-28 08:56:10 | INFO     | DF | Enhanced CBAK: 3.3116347789764404
2024-03-28 08:56:10 | INFO     | DF | Enhanced COVL: 3.472588062286377
2024-03-28 08:56:10 | INFO     | DF | Enhanced SSNR: 7.947866439819336
2024-03-28 08:56:10 | INFO     | DF | Enhanced SISDR: 16.64629923125494
0.9420673982468806,2.809007406234741,4.124786853790283,3.3116347789764404,3.472588062286377,16.64629923125494
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet2/
2024-03-28 08:58:36 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:58:36 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:58:36 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:58:36 | INFO     | DF | Loading model settings of DeepFilterNet2
2024-03-28 08:58:36 | INFO     | DF | Initializing model `deepfilternet2`
2024-03-28 08:58:36 | INFO     | DF | Found checkpoint models/DeepFilterNet2/checkpoints/model_96.ckpt.best with epoch 96
2024-03-28 08:58:36 | INFO     | DF | Running on device cpu
2024-03-28 08:58:36 | INFO     | DF | Model loaded
2024-03-28 08:59:46 | INFO     | DF | Progress: 25%
2024-03-28 09:00:55 | INFO     | DF | Progress: 50%
2024-03-28 09:02:06 | INFO     | DF | Progress: 75%
2024-03-28 09:03:19 | INFO     | DF | Progress: 100%
2024-03-28 09:03:19 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 09:03:20 | INFO     | DF | Enhanced STOI: 0.8134182890036015
2024-03-28 09:03:20 | INFO     | DF | Enhanced PESQ: 2.714287281036377
2024-03-28 09:03:20 | INFO     | DF | Enhanced CSIG: 3.8609461784362793
2024-03-28 09:03:20 | INFO     | DF | Enhanced CBAK: 2.533511161804199
2024-03-28 09:03:20 | INFO     | DF | Enhanced COVL: 3.253587484359741
2024-03-28 09:03:20 | INFO     | DF | Enhanced SSNR: -2.2124998569488525
2024-03-28 09:03:20 | INFO     | DF | Enhanced SISDR: -22.458605475868413
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet3/
2024-03-28 08:40:25 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:40:25 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:40:25 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:40:25 | INFO     | DF | Loading model settings of DeepFilterNet3
2024-03-28 08:40:25 | INFO     | DF | Initializing model `deepfilternet3`
2024-03-28 08:40:25 | INFO     | DF | Found checkpoint models/DeepFilterNet3/checkpoints/model_120.ckpt.best with epoch 120
2024-03-28 08:40:25 | INFO     | DF | Running on device cpu
2024-03-28 08:40:25 | INFO     | DF | Model loaded
2024-03-28 08:41:33 | INFO     | DF | Progress: 25%
2024-03-28 08:42:42 | INFO     | DF | Progress: 50%
2024-03-28 08:43:52 | INFO     | DF | Progress: 75%
2024-03-28 08:45:04 | INFO     | DF | Progress: 100%
2024-03-28 08:45:04 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:45:05 | INFO     | DF | Enhanced STOI: 0.6192451893910225
2024-03-28 08:45:05 | INFO     | DF | Enhanced PESQ: 3.1094107627868652
2024-03-28 08:45:05 | INFO     | DF | Enhanced CSIG: 3.632720708847046
2024-03-28 08:45:05 | INFO     | DF | Enhanced CBAK: 2.6275570392608643
2024-03-28 08:45:05 | INFO     | DF | Enhanced COVL: 3.30583119392395
2024-03-28 08:45:05 | INFO     | DF | Enhanced SSNR: -2.231890916824341
2024-03-28 08:45:05 | INFO     | DF | Enhanced SISDR: -24.651835330047653
0.6192451893910225,3.1094107627868652,3.632720708847046,2.6275570392608643,3.30583119392395,-24.651835330047653

my_requirements.txt
@QQQQQQQQY
Copy link

I am currently on last main commit f2445da

I downloaded Voicebank Demand test set from here I evaluated DeepFilterNet versions 1-3 using the test_voicebank_demand.py script.

Results of DeepFilterNet1 are the same as those reported in the DFN3 paper However, the results of DFN2 and 3 and are much lower. image

How I can reproduce results similar to those in the paper?

python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet/
2024-03-28 08:47:53 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:47:53 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:47:53 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:47:53 | INFO     | DF | Loading model settings of DeepFilterNet
2024-03-28 08:47:53 | INFO     | DF | Initializing model `deepfilternet`
2024-03-28 08:47:53 | WARNING  | DF | Could not find `best` checkpoint. Checking for default...
2024-03-28 08:47:53 | INFO     | DF | Found checkpoint models/DeepFilterNet/checkpoints/model_30.ckpt with epoch 30
2024-03-28 08:47:53 | INFO     | DF | Running on device cpu
2024-03-28 08:47:53 | INFO     | DF | Model loaded
2024-03-28 08:49:56 | INFO     | DF | Progress: 25%
2024-03-28 08:52:02 | INFO     | DF | Progress: 50%
2024-03-28 08:54:06 | INFO     | DF | Progress: 75%
2024-03-28 08:56:09 | INFO     | DF | Progress: 100%
2024-03-28 08:56:09 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:56:10 | INFO     | DF | Enhanced STOI: 0.9420673982468806
2024-03-28 08:56:10 | INFO     | DF | Enhanced PESQ: 2.809007406234741
2024-03-28 08:56:10 | INFO     | DF | Enhanced CSIG: 4.124786853790283
2024-03-28 08:56:10 | INFO     | DF | Enhanced CBAK: 3.3116347789764404
2024-03-28 08:56:10 | INFO     | DF | Enhanced COVL: 3.472588062286377
2024-03-28 08:56:10 | INFO     | DF | Enhanced SSNR: 7.947866439819336
2024-03-28 08:56:10 | INFO     | DF | Enhanced SISDR: 16.64629923125494
0.9420673982468806,2.809007406234741,4.124786853790283,3.3116347789764404,3.472588062286377,16.64629923125494
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet2/
2024-03-28 08:58:36 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:58:36 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:58:36 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:58:36 | INFO     | DF | Loading model settings of DeepFilterNet2
2024-03-28 08:58:36 | INFO     | DF | Initializing model `deepfilternet2`
2024-03-28 08:58:36 | INFO     | DF | Found checkpoint models/DeepFilterNet2/checkpoints/model_96.ckpt.best with epoch 96
2024-03-28 08:58:36 | INFO     | DF | Running on device cpu
2024-03-28 08:58:36 | INFO     | DF | Model loaded
2024-03-28 08:59:46 | INFO     | DF | Progress: 25%
2024-03-28 09:00:55 | INFO     | DF | Progress: 50%
2024-03-28 09:02:06 | INFO     | DF | Progress: 75%
2024-03-28 09:03:19 | INFO     | DF | Progress: 100%
2024-03-28 09:03:19 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 09:03:20 | INFO     | DF | Enhanced STOI: 0.8134182890036015
2024-03-28 09:03:20 | INFO     | DF | Enhanced PESQ: 2.714287281036377
2024-03-28 09:03:20 | INFO     | DF | Enhanced CSIG: 3.8609461784362793
2024-03-28 09:03:20 | INFO     | DF | Enhanced CBAK: 2.533511161804199
2024-03-28 09:03:20 | INFO     | DF | Enhanced COVL: 3.253587484359741
2024-03-28 09:03:20 | INFO     | DF | Enhanced SSNR: -2.2124998569488525
2024-03-28 09:03:20 | INFO     | DF | Enhanced SISDR: -22.458605475868413
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet3/
2024-03-28 08:40:25 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:40:25 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:40:25 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:40:25 | INFO     | DF | Loading model settings of DeepFilterNet3
2024-03-28 08:40:25 | INFO     | DF | Initializing model `deepfilternet3`
2024-03-28 08:40:25 | INFO     | DF | Found checkpoint models/DeepFilterNet3/checkpoints/model_120.ckpt.best with epoch 120
2024-03-28 08:40:25 | INFO     | DF | Running on device cpu
2024-03-28 08:40:25 | INFO     | DF | Model loaded
2024-03-28 08:41:33 | INFO     | DF | Progress: 25%
2024-03-28 08:42:42 | INFO     | DF | Progress: 50%
2024-03-28 08:43:52 | INFO     | DF | Progress: 75%
2024-03-28 08:45:04 | INFO     | DF | Progress: 100%
2024-03-28 08:45:04 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:45:05 | INFO     | DF | Enhanced STOI: 0.6192451893910225
2024-03-28 08:45:05 | INFO     | DF | Enhanced PESQ: 3.1094107627868652
2024-03-28 08:45:05 | INFO     | DF | Enhanced CSIG: 3.632720708847046
2024-03-28 08:45:05 | INFO     | DF | Enhanced CBAK: 2.6275570392608643
2024-03-28 08:45:05 | INFO     | DF | Enhanced COVL: 3.30583119392395
2024-03-28 08:45:05 | INFO     | DF | Enhanced SSNR: -2.231890916824341
2024-03-28 08:45:05 | INFO     | DF | Enhanced SISDR: -24.651835330047653
0.6192451893910225,3.1094107627868652,3.632720708847046,2.6275570392608643,3.30583119392395,-24.651835330047653

my_requirements.txt

That's what I'm getting. Did you manage to fix it?

@WoodieDudy
Copy link
Author

I have updated torch to 2.2 and that fixed problem.
There's is my new requirements
reqs.txt
But I haven't figured out what exactly was causing the problem yet.
I'll text when I find out.

@QQQQQQQQY
Copy link

我已将火炬更新到 2.2 并解决了这个问题。这是我的新要求reqs.txt但我还没有弄清楚到底是什么导致了这个问题。当我发现时，我会发短信。

I have updated torch to 2.2 and that fixed problem. There's is my new requirements reqs.txt But I haven't figured out what exactly was causing the problem yet. I'll text when I find out.

I appreciate you writing back. I'll try it.

@QQQQQQQQY
Copy link

我已将火炬更新到 2.2 并解决了这个问题。这是我的新要求reqs.txt但我还没有弄清楚到底是什么导致了这个问题。当我发现时，我会发短信。

I have updated torch to 2.2 and that fixed problem. There's is my new requirements reqs.txt But I haven't figured out what exactly was causing the problem yet. I'll text when I find out.
Hi, I upgraded torch=2.2.2. the last result of Deepfilter2 is as follows:
2024-04-04 16:45:36 | INFO | DF | Running on torch 2.2.2+cu118
2024-04-04 16:45:36 | INFO | DF | Running on host DESKTOP-J0UR71S
2024-04-04 16:45:36 | INFO | DF | Loading model settings of outputs
2024-04-04 16:45:36 | INFO | DF | Initializing model deepfilternet2
2024-04-04 16:45:38 | INFO | DF | Found checkpoint D:\DeepFilterNet-main\DeepFilterNet\outputs\checkpoints\model_60.ckpt.best with epoch 60
2024-04-04 16:45:38 | INFO | DF | Running on device cuda:0
2024-04-04 16:45:38 | INFO | DF | Model loaded
2024-04-04 16:45:53 | INFO | DF | Progress: 25%
2024-04-04 16:46:08 | INFO | DF | Progress: 50%
2024-04-04 16:46:22 | INFO | DF | Progress: 75%
2024-04-04 16:46:36 | INFO | DF | Progress: 100%
2024-04-04 16:46:36 | INFO | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-04-04 16:48:02 | INFO | DF | Enhanced STOI: 0.925880108458049
2024-04-04 16:48:02 | INFO | DF | Enhanced PESQ: 2.823953866958618
2024-04-04 16:48:02 | INFO | DF | Enhanced CSIG: 4.110456466674805
2024-04-04 16:48:02 | INFO | DF | Enhanced CBAK: 3.1988513469696045
2024-04-04 16:48:02 | INFO | DF | Enhanced COVL: 3.4739575386047363
2024-04-04 16:48:02 | INFO | DF | Enhanced SSNR: 6.198051452636719
2024-04-04 16:48:02 | INFO | DF | Enhanced SISDR: 13.75959416670562
It still doesn't achieve the results in the original article, so may I ask if you succeeded? I found your best epoch at 96 while I'm at 60, can you share your config.ini? I'm very much looking forward to your reply

@WoodieDudy
Copy link
Author

I took weights and config.ini from there

@satvik-venkatesh
Copy link

@WoodieDudy upgrading torch to 2.2 surprisingly worked!! I also installed pystoi and pesq package versions from your requirements file. Thank you.

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the stale label Jul 10, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@satvik-venkatesh @WoodieDudy @QQQQQQQQY