[AutoProcessor] Add Wav2Vec2WithLM & small fix #14675
Conversation
| @@ -39,6 +39,7 @@ | |||
| ("speech_to_text_2", "Speech2Text2Processor"), | |||
| ("trocr", "TrOCRProcessor"), | |||
| ("wav2vec2", "Wav2Vec2Processor"), | |||
| ("wav2vec2_with_lm", "Wav2Vec2ProcessorWithLM"), | |||
There was a problem hiding this comment.
need new folder for this
| @@ -145,6 +146,9 @@ def from_pretrained(cls, pretrained_model_name_or_path, **kwargs): | |||
| key: kwargs[key] for key in ["revision", "use_auth_token", "local_files_only"] if key in kwargs | |||
| } | |||
| model_files = get_list_of_files(pretrained_model_name_or_path, **get_list_of_files_kwargs) | |||
| # strip to file name | |||
| model_files = [f.split("/")[-1] for f in model_files] | |||
There was a problem hiding this comment.
Shouldn't that be handled in get_list_of_files?
There was a problem hiding this comment.
yeah I thought so too in the beginning, but think it's cleaner to get the actual file names from this function instead of trimming to the last file. What do you think? cc @sgugger as well
There was a problem hiding this comment.
It should be done at the model_files level to be consistent with the distant repos.
|
Test failure is unrelated |
|
Failures are unrelated - merging |
There was a problem hiding this comment.
Can't comment on the merged PR but I think we should strip the filenames in the get_list_of_files method to have a behavior that is consistent between repos and local folders. The files are not directly usable and you have to use cached_path anyway to actually use them, which will put again the folder name at the beginning.
| @@ -145,6 +146,9 @@ def from_pretrained(cls, pretrained_model_name_or_path, **kwargs): | |||
| key: kwargs[key] for key in ["revision", "use_auth_token", "local_files_only"] if key in kwargs | |||
| } | |||
| model_files = get_list_of_files(pretrained_model_name_or_path, **get_list_of_files_kwargs) | |||
| # strip to file name | |||
| model_files = [f.split("/")[-1] for f in model_files] | |||
There was a problem hiding this comment.
It should be done at the model_files level to be consistent with the distant repos.
* [AutoProcessor] Add Wav2Vec2WithLM & small fix * revert line removal * Update src/transformers/__init__.py * add test * up * up * small fix
What does this PR do?
Make AutoProcessor work correctly with local files and small fix
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.