Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* s2t * fix config * conversion script * fix import * add tokenizer * fix tok init * fix tokenizer * first version working * fix embeds * fix lm head * remove extra heads * fix convert script * handle encoder attn mask * style * better enc attn mask * override _prepare_attention_mask_for_generation * handle attn_maks in encoder and decoder * input_ids => input_features * enable use_cache * remove old code * expand embeddings if needed * remove logits bias * masked_lm_loss => loss * hack tokenizer to support feature processing * fix model_input_names * style * fix error message * doc * remove inputs_embeds * remove input_embeds * remove unnecessary docstring * quality * SpeechToText => Speech2Text * style * remove shared_embeds * subsample => conv * remove Speech2TextTransformerDecoderWrapper * update output_lengths formula * fix table * remove max_position_embeddings * update conversion scripts * add possibility to do upper case for now * add FeatureExtractor and Processor * add tests for extractor * require_torch_audio => require_torchaudio * add processor test * update import * remove classification head * attention mask is now 1D * update docstrings * attention mask should be of type long * handle attention mask from generate * alwyas return attention_mask * fix test * style * doc * Speech2TextTransformer => Speech2Text * Speech2TextTransformerConfig => Speech2TextConfig * remove dummy_inputs * nit * style * multilinguial tok * fix tokenizer * add tgt_lang setter * save lang_codes * fix tokenizer * add forced_bos_token_id to tokenizer * apply review suggestions * add torchaudio to extra deps * add speech deps to CI * fix dep * add libsndfile to ci * libsndfile1 * add speech to extras all * libsndfile1 -> libsndfile1 * libsndfile * libsndfile1-dev * apt update * add sudo to install * update deps table * install libsndfile1-dev on CI * tuple to list * init conv layer * add model tests * quality * add integration tests * skip_special_tokens * add speech_to_text_transformer in toctree * fix tokenizer * fix fp16 tests * add tokenizer tests * fix copyright * input_values => input_features * doc * add model in readme * doc * change checkpoint names * fix copyright * fix code example * add max_model_input_sizes in tokenizer * fix integration tests * add do_lower_case to tokenizer * remove clamp trick * fix "Add modeling imports here" * fix copyrights * fix tests * SpeechToTextTransformer => SpeechToText * fix naming * fix table formatting * fix typo * style * fix typos * remove speech dep from extras[testing] * fix copies * rename doc file, * put imports under is_torch_available * run feat extract tests when torch is available * dummy objects for processor and extractor * fix imports in tests * fix import in modeling test * fxi imports * fix torch import * fix imports again * fix positional embeddings * fix typo in import * adapt new extractor refactor * style * fix torchscript test * doc * doc * Apply suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * fix docs, copied from, style * fix docstring * handle imports * remove speech from all extra deps * remove s2t from seq2seq lm mapping * better names * skip training tests * add install instructions * List => Tuple * doc * fix conversion script * fix urls * add instruction for libsndfile * fix fp16 test Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
- Loading branch information
3 people
committed
Mar 10, 2021
1 parent
efb5c0a
commit d26b37e
Showing
30 changed files
with
3,833 additions
and
24 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| .. | ||
| Copyright 2021 The HuggingFace Team. All rights reserved. | ||
| Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
| an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations under the License. | ||
|
|
||
| Speech2Text | ||
| ----------------------------------------------------------------------------------------------------------------------- | ||
|
|
||
| Overview | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The Speech2Text model was proposed in `fairseq S2T: Fast Speech-to-Text Modeling with fairseq | ||
| <https://arxiv.org/abs/2010.05171>`__ by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It's a | ||
| transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech | ||
| Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are | ||
| fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the | ||
| transcripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST: | ||
| `LibriSpeech <http://www.openslr.org/12>`__, `CoVoST 2 <https://github.com/facebookresearch/covost>`__, `MuST-C | ||
| <https://ict.fbk.eu/must-c/>`__. | ||
|
|
||
| The original code can be found `here <https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text>`__. | ||
|
|
||
|
|
||
| Inference | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech | ||
| signal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The | ||
| :obj:`generate()` method can be used for inference. | ||
|
|
||
| The :class:`~transformers.Speech2TextFeatureExtractor` class is responsible for extracting the log-mel filter-bank | ||
| features. The :class:`~transformers.Speech2TextProcessor` wraps :class:`~transformers.Speech2TextFeatureExtractor` and | ||
| :class:`~transformers.Speech2TextTokenizer` into a single instance to both extract the input features and decode the | ||
| predicted token ids. | ||
|
|
||
| The feature extractor depends on :obj:`torchaudio` and the tokenizer depends on :obj:`sentencepiece` so be sure to | ||
| install those packages before running the examples. You could either install those as extra speech dependancies with | ||
| ``pip install transformers"[speech, sentencepiece]"`` or install the packages seperatly with ``pip install torchaudio | ||
| sentencepiece``. Also ``torchaudio`` requires the development version of the `libsndfile | ||
| <http://www.mega-nerd.com/libsndfile/>`__ package which can be installed via a system package manager. On Ubuntu it can | ||
| be installed as follows: ``apt install libsndfile1-dev`` | ||
|
|
||
|
|
||
| - ASR and Speech Translation | ||
|
|
||
| .. code-block:: | ||
| >>> import torch | ||
| >>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration | ||
| >>> from datasets import load_dataset | ||
| >>> import soundfile as sf | ||
| >>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr") | ||
| >>> processor = Speech2Textprocessor.from_pretrained("facebook/s2t-small-librispeech-asr") | ||
| >>> def map_to_array(batch): | ||
| ... speech, _ = sf.read(batch["file"]) | ||
| ... batch["speech"] = speech | ||
| ... return batch | ||
| >>> ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") | ||
| >>> ds = ds.map(map_to_array) | ||
| >>> input_features = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt").input_features # Batch size 1 | ||
| >>> generated_ids = model.generate(input_ids=input_features) | ||
| >>> transcription = processor.batch_decode(generated_ids) | ||
| - Multilingual speech translation | ||
|
|
||
| For multilingual speech translation models, :obj:`eos_token_id` is used as the :obj:`decoder_start_token_id` and | ||
| the target language id is forced as the first generated token. To force the target language id as the first | ||
| generated token, pass the :obj:`forced_bos_token_id` parameter to the :obj:`generate()` method. The following | ||
| example shows how to transate English speech to French text using the `facebook/s2t-medium-mustc-multilingual-st` | ||
| checkpoint. | ||
|
|
||
| .. code-block:: | ||
| >>> import torch | ||
| >>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration | ||
| >>> from datasets import load_dataset | ||
| >>> import soundfile as sf | ||
| >>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-medium-mustc-multilingual-st") | ||
| >>> processor = Speech2Textprocessor.from_pretrained("facebook/s2t-medium-mustc-multilingual-st") | ||
| >>> def map_to_array(batch): | ||
| ... speech, _ = sf.read(batch["file"]) | ||
| ... batch["speech"] = speech | ||
| ... return batch | ||
| >>> ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") | ||
| >>> ds = ds.map(map_to_array) | ||
| >>> input_features = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt").input_features # Batch size 1 | ||
| >>> generated_ids = model.generate(input_ids=input_features, forced_bos_token_id=processor.tokenizer.lang_code_to_id["fr"]) | ||
| >>> translation = processor.batch_decode(generated_ids) | ||
| See the `model hub <https://huggingface.co/models?filter=speech_to_text>`__ to look for Speech2Text checkpoints. | ||
|
|
||
|
|
||
| Speech2TextConfig | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextConfig | ||
| :members: | ||
|
|
||
|
|
||
| Speech2TextTokenizer | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextTokenizer | ||
| :members: build_inputs_with_special_tokens, get_special_tokens_mask, | ||
| create_token_type_ids_from_sequences, save_vocabulary | ||
|
|
||
|
|
||
| Speech2TextFeatureExtractor | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextFeatureExtractor | ||
| :members: __call__ | ||
|
|
||
|
|
||
| Speech2TextProcessor | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextProcessor | ||
| :members: __call__, from_pretrained, save_pretrained, batch_decode, decode, as_target_processor | ||
|
|
||
|
|
||
| Speech2TextModel | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextModel | ||
| :members: forward | ||
|
|
||
|
|
||
| Speech2TextForConditionalGeneration | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| .. autoclass:: transformers.Speech2TextForConditionalGeneration | ||
| :members: forward |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.