Training your own data set (audio and transcription)

[tumusudheer]

Hi,

I would like to train my own dataset. Just wanted to know some guidelines on how to prepare data:
I'll prepare some .wav files with 16KHz sample rate and single channel.

1. What should be length of each audio file (in terms of seconds/milli seconds) ?
2. Is there a max and min length of an audio file can be (in terms of duration) ?
3. Also is there a limit on number of words/characters that a single audio file's transcription should be ?

Also is a there a trained model I can start my training from instead of starting from scratch ?

Thanks.

[tumusudheer]

Also I trained the model with just 4 audio files of common_voice data , to give it a try. It has prepared encoder.subwords file and the vocab size is given as 301. I trained the model for 1000 epochs.

Now I want to train first 1000 audio files. When I used the command
python run_rnnt.py --mode train --data_dir data_trail/preprocessed_small_1000 --batch_size 8 --eval_size 100 --checkpoint model.bak.small_size_data/checkpoint_1000_826.4307.hdf5, I'm getting the following error:

I believe this is due to new words got added; But for finetuing, the model is copying the old encounter.subwords from the checkpoint directory as here

When we want to to train with new data and want to finetune (restore weights from a trained checkpoint), how should we add new vocabulary of the new audio files transcriptions.

2020-06-12 15:05:46.302226: W tensorflow/core/framework/op_kernel.cc:1741] Invalid argument: ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/tmp/tmpp084qsez.py", line 12, in <lambda>
    retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
    return _call_unconverted(f, args, kwargs, options, False)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
    return f(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
    return tf.constant(encoder.decode(ids.numpy()))

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
    subword = self._id_to_subword(subword_id)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
    "[0, %d)." % (subword_id + 1, self.vocab_size))

ValueError: Received id 502 which is invalid. Ids must be within [0, 301).


Traceback (most recent call last):
  File "run_rnnt.py", line 590, in <module>
    app.run(main)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_rnnt.py", line 549, in main
    eval_metrics=[accuracy_fn, wer_fn])
  File "run_rnnt.py", line 347, in run_training
    checkpoint_model()
  File "run_rnnt.py", line 312, in checkpoint_model
    metrics=eval_metrics)
  File "run_rnnt.py", line 436, in run_evaluate
    loss, metrics_results = eval_step(inputs)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 650, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/tmp/tmpp084qsez.py", line 12, in <lambda>
    retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
    return _call_unconverted(f, args, kwargs, options, False)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
    return f(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
    return tf.constant(encoder.decode(ids.numpy()))

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
    subword = self._id_to_subword(subword_id)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
    "[0, %d)." % (subword_id + 1, self.vocab_size))

ValueError: Received id 502 which is invalid. Ids must be within [0, 301).


         [[node EagerPyFunc (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py:40) ]]
  (1) Invalid argument:  ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/tmp/tmpp084qsez.py", line 12, in <lambda>
    retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
    return _call_unconverted(f, args, kwargs, options, False)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
    return f(*args)

  File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
    return tf.constant(encoder.decode(ids.numpy()))

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
    subword = self._id_to_subword(subword_id)

  File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
    "[0, %d)." % (subword_id + 1, self.vocab_size))

ValueError: Received id 502 which is invalid. Ids must be within [0, 301).


         [[node EagerPyFunc (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py:40) ]]
         [[StringSplit_1/StringSplit/StringSplitV2/_130]]
0 successful operations.
0 derived errors ignored. [Op:__inference_eval_step_5969]

Errors may have originated from an input operation.
Input Source operations connected to node EagerPyFunc:
 strided_slice_4 (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/metrics.py:81)

Input Source operations connected to node EagerPyFunc:
 strided_slice_4 (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/metrics.py:81)

Function call stack:
eval_step -> eval_step