Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to Train Google Speech Dataset #228

Open
implosion07 opened this issue Jul 15, 2023 · 6 comments
Open

Unable to Train Google Speech Dataset #228

implosion07 opened this issue Jul 15, 2023 · 6 comments
Labels
bug Something isn't working

Comments

@implosion07
Copy link

What happened + What you expected to happen

We are unable to train on the google speech dataset. While training, we get the error as shown below
Please help us out.

Screenshot from 2023-07-15 12-13-29

Versions / Dependencies

We are running on Red Hat servers and are using python 3.7

Reproduction script

It seems like the problem is with resnet code for it is error due to mismatch in the number of channels.

Issue Severity

High: It blocks me from completing my task.

@implosion07 implosion07 added the bug Something isn't working label Jul 15, 2023
@fanlai0990
Copy link
Member

Thanks for trying FedScale. Note that the model you are using perhaps is from torchvision. Please refer to these specialized models for speech task: https://github.com/SymbioticLab/FedScale/blob/master/fedscale/cloud/fllibs.py#L129-L162

@implosion07
Copy link
Author

Thanks for the reply @fanlai0990 .
However we are using the same model as provided in the repository. Also we are using the same configurations as provided in the config file: https://github.com/SymbioticLab/FedScale/blob/master/benchmark/configs/speech/google_speech.yml
Especially the task which is set to speech.

@PanPapag
Copy link

Hello, I'm encountering the same problem mentioned by @implosion07 . I've used the provided model from the repository and followed the configurations exactly as specified in the config file https://github.com/SymbioticLab/FedScale/blob/master/benchmark/configs/speech/google_speech.yml . Unfortunately, I haven't been able to resolve the issue. Just wanted to add my experience here to raise awareness about the problem.
Thanks in advance!

@fanlai0990
Copy link
Member

Thanks for letting us know. I just finished my test run, but it seems the code works well for me. Did you install FedScale from the source code?
Screen Shot 2023-07-19 at 4 16 48 PM

@implosion07
Copy link
Author

implosion07 commented Jul 19, 2023

Thanks for checking @fanlai0990 We are using the following configuration. Can you help us in finding out the mistake if any.
Screenshot from 2023-07-19 19-11-00

@fanlai0990
Copy link
Member

It's hard for us to identify the issue as we can not reproduce it. For now, (1) it seems the data_dir is slightly different to the default one. Please double check it; (2) roll back to the default configuration here to check it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants