-
Notifications
You must be signed in to change notification settings - Fork 77
Conversation
for more information, see https://pre-commit.ci
hey @SeanNaren just made the PR, would be great if you could take a look and share some insights :) |
Hi @rafaelvp-db thanks for making the PR! There are quite a few pieces missing, hopefully, I can assist in helping you get the right things implemented! Firstly I don't think your current code is entirely correct (unless I've made a mistake). The Wav2Vec model/dataset/tokenizer are completely different and should probably exist as new classes inherited from What I think would be a good idea is to get this blog post implemented into a TaskTransformer and a TransformerDataModule as you've already outlined: https://huggingface.co/blog/fine-tune-wav2vec2-english This would involve
Overall I would assume something like this to work: import pytorch_lightning as pl
from lightning_transformers.task.audio.speech_recognition import (
SpeechRecognitionDataConfig,
SpeechRecognitionDataModule,
SpeechRecognitionTransformer,
)
if __name__ == "__main__":
model = SpeechRecognitionTransformer("facebook/wav2vec2-base", ctc_loss_reduction="mean", vocab_file="vocab.json")
dm = SpeechRecognitionDataModule(
cfg=SpeechRecognitionDataConfig(
batch_size=1,
dataset_name="timit_asr",
),
tokenizer=model.tokenizer,
)
trainer = pl.Trainer(accelerator="auto", devices="auto", max_epochs=1)
trainer.fit(model, dm) |
Thanks for the guidelines @SeanNaren! Let me look into that. |
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Codecov Report
@@ Coverage Diff @@
## master #251 +/- ##
=====================================
- Coverage 75% 74% -1%
=====================================
Files 73 77 +4
Lines 1622 1682 +60
=====================================
+ Hits 1210 1245 +35
- Misses 412 437 +25 |
Hows it going @rafaelvp-db? The code looks muuuch nicer, amazing job! Anything I can assist with? I notice that the example requires a vocab.json, I'm sure we can YOLO it and use the alphabet. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Initial PR for ASR support