You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm a bit confused after reading changes from #58. I was currently using token_text for my work, and I would prefer to continue, if it is possible, using token_text instead of text (because I use special tags similar to <space>).
After reading changes from #58 I have the impression that it's still possible to use token_text as outputs for ASR systems, is it right?
If so, what argument should be given to the training script?
Which no additional argument, when training with JSON that includes token_text instead of text, I keep getting this error:
Traceback (most recent call last):
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/export/home/lium/vpelloin/git/espresso/fairseq/distributed/utils.py", line 328, in distributed_main
main(cfg, **kwargs)
File "/export/home/lium/vpelloin/git/espresso/fairseq_cli/train.py", line 176, in main
valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/contextlib.py", line 74, in inner
return func(*args, **kwds)
File "/export/home/lium/vpelloin/git/espresso/fairseq_cli/train.py", line 287, in train
log_output = trainer.train_step(samples)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/contextlib.py", line 74, in inner
return func(*args, **kwds)
File "/export/home/lium/vpelloin/git/espresso/fairseq/trainer.py", line 674, in train_step
ignore_grad=is_dummy_batch,
File "/export/home/lium/vpelloin/git/espresso/fairseq/tasks/fairseq_task.py", line 476, in train_step
loss, sample_size, logging_output = criterion(model, sample)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/export/home/lium/vpelloin/git/espresso/espresso/criterions/label_smoothed_cross_entropy_v2.py", line 150, in forward
net_output = model(**sample["net_input"], epoch=self.epoch)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/export/home/lium/vpelloin/git/espresso/fairseq/distributed/module_proxy_wrapper.py", line 55, in forward
return self.module(*args, **kwargs)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/export/home/lium/vpelloin/git/espresso/fairseq/distributed/legacy_distributed_data_parallel.py", line 74, in forward
return self.module(*inputs, **kwargs)
File "/lium/home/vpelloin/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'prev_output_tokens'
Thank you so much for the incredible work you're doing with this tool!
The text was updated successfully, but these errors were encountered:
Maybe you can update the field name token_text to text in the json files, and then remove the arg --bpe sentencepiece --sentencepiece-model ${sentencepiece_model}.model passed to train.py. This will take the text as it-is without any v sentencepiece encoding process.
edit: there is a later commit 4c86e23 doing on-the-fly tokenization, where token_text is totally removed from the code. If you are using the version after that, I see where your confusion is from
By renaming token_text to text, and removing the --bpe argument, fairseq dit not complain, and I achieved to train my model.
Now, I'm just having trouble when decoding, as there is no difference between the WER and the CER metric (decoded_char_results and decoded_results are the same). I think creating my own tokeniser/bpe class should work, I might try that later. Otherwise, I'll look the decoding functions and add some custom code.
❓ Questions and Help
Hello,
I'm a bit confused after reading changes from #58. I was currently using token_text for my work, and I would prefer to continue, if it is possible, using token_text instead of text (because I use special tags similar to
<space>
).After reading changes from #58 I have the impression that it's still possible to use token_text as outputs for ASR systems, is it right?
If so, what argument should be given to the training script?
Which no additional argument, when training with JSON that includes token_text instead of text, I keep getting this error:
Thank you so much for the incredible work you're doing with this tool!
The text was updated successfully, but these errors were encountered: