Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot download models locally #1070

Open
Aksh97 opened this issue Oct 27, 2021 · 10 comments · May be fixed by #2476
Open

Cannot download models locally #1070

Aksh97 opened this issue Oct 27, 2021 · 10 comments · May be fixed by #2476
Assignees
Projects

Comments

@Aksh97
Copy link

Aksh97 commented Oct 27, 2021

Hi, I tried to run the commands, which worked perfectly file.


from speechbrain.pretrained import EncoderDecoderASR

asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-crdnn-rnnlm-librispeech")

But when I looked over the model files, the files downloaded are just 158 kbs, and not the full files are downloaded.

Also, when I tried to download the files directly from Files and versions, the ckpt files are downloaded as zip. which cant be used later for inference. IS there any way to resolve it or it has been purposefully made like that.

@Gastron
Copy link
Collaborator

Gastron commented Oct 27, 2021

The full files are downloaded, but if they're from HuggingFace Hub like here, the full files go the HuggingFace cache, and just symlinks are created, is this what you're actually seeing?

This was just recently discussed in #1055. As this surprises people, maybe we should change it. The HuggingFace cache is the standard location for HuggingFace downloads and the point of a cache is of course to avoid multiple copies of the same files. I'd like to know, though, what do you want to do with the full files - is it not sufficient to have symlinks?

@Aksh97
Copy link
Author

Aksh97 commented Oct 29, 2021

Hi @Gastron, thanks for the reply. I wanted to store those files and containerize them so that it does not needs to be downloaded whenever I run the container.
But unlike other repositories, where the full files are downloaded by default, instead of cache. Maybe for speechbrain we can put an arguement symlinks=true, to just add the symlinks and keep that by default. But if we need that to download full files, we can pass the arguement symlinks=false, we can at least download them.

@mravanelli
Copy link
Collaborator

mravanelli commented Oct 29, 2021 via email

@Gastron
Copy link
Collaborator

Gastron commented Oct 29, 2021

Right, noted.

@Aksh97
Copy link
Author

Aksh97 commented Nov 3, 2021

Thanks @Gastron and @mravanelli For the consideration.

@Aksh97 Aksh97 closed this as completed Nov 15, 2021
@Aksh97
Copy link
Author

Aksh97 commented Nov 17, 2021

Hi There, Any further improvement or solution for this or it will take some time to implement the changes?

@Aksh97 Aksh97 reopened this Nov 17, 2021
@mravanelli
Copy link
Collaborator

@Gastron, do we want to change the download folder in the end?

@Gastron
Copy link
Collaborator

Gastron commented Nov 17, 2021

Let's try to briefly discuss in the SpeechBrain core-team meeting today.

@MClarkTurner
Copy link

Since, this issue is still open and it has been about 4 months. I also wanted to comment that the current implementation is unintuitive. This is especially egregious given the standard set by PyTorch, upon which speechbrain is built and what an outsider would consider, at a glance, to be standard practice. Furthermore, it was only after stumbling upon this issue board and the comments in #1055 that I was able to resolve changing the location of the cache placement (which is an otherwise undocumented process). For those interested, the solution is the following code:

export HUGGINGFACE_HUB_CACHE=<new_directory>

otherwise the code will default to placing a ./cache/huggingface under your /home directory.

@anautsch anautsch added this to To do in CI/CD via automation Apr 21, 2022
@anautsch anautsch moved this from To do to Performance & housekeeping in CI/CD Apr 21, 2022
@Adel-Moumen
Copy link
Collaborator

Hello,

Is the issue still up? We merged a PR #1817 which is modifying a lot of things related to HuggingFace, Pretrainer and so on. Could you please let me know if now everything is working as intended?

Thanks.

@asumagic asumagic linked a pull request Mar 27, 2024 that will close this issue
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
CI/CD
Performance & housekeeping
Development

Successfully merging a pull request may close this issue.

5 participants