Bus error: 10 at training

[kouohhashi]

Hi,
My question could be stupid but I could not run python train.py on my mac

Here's logs on terminal.

When I use prepro = False,

(tacotron) MacBook-Air:tacotron me$ python train.py 
Training Graph loaded
  0%|                                          | 0/343 [00:00<?, ?b/s]Bus error: 10

When I use prepro = True,

(tacotron) MacBook-Air:tacotron me$ python train.py 
Training Graph loaded
  0%|                                          | 0/343 [00:00<?, ?b/s]2018-09-08 11:14:51.345998: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.598145: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.598416: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.598518: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.598628: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.600275: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.600851: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.601028: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.601194: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.602308: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.602480: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.602747: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:51.684756: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:53.715177: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:54.001989: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
2018-09-08 11:14:54.287296: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):                                    
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_bucket_by_sequence_length/bucket/top_queue' is closed and has insufficient elements (requested 1, current size 0)
   [[Node: bucket_by_sequence_length/bucket/dequeue_top = QueueDequeueV2[component_types=[DT_INT32, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](bucket_by_sequence_length/bucket/top_queue)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 954, in managed_session
    yield sess
  File "train.py", line 103, in <module>
    _, gs = sess.run([g.train_op, g.global_step])
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_bucket_by_sequence_length/bucket/top_queue' is closed and has insufficient elements (requested 1, current size 0)
   [[Node: bucket_by_sequence_length/bucket/dequeue_top = QueueDequeueV2[component_types=[DT_INT32, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](bucket_by_sequence_length/bucket/top_queue)]]

Caused by op 'bucket_by_sequence_length/bucket/dequeue_top', defined at:
  File "train.py", line 96, in <module>
    g = Graph(); print("Training Graph loaded")
  File "train.py", line 33, in __init__
    self.x, self.y, self.z, self.fnames, self.num_batch = get_batch()
  File "/Volumes/AI/TTS/tacotron/data_load.py", line 117, in get_batch
    dynamic_pad=True)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/contrib/training/python/training/bucket_ops.py", line 414, in bucket_by_sequence_length
    shared_name=shared_name)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/contrib/training/python/training/bucket_ops.py", line 288, in bucket
    dequeued = top_queue.dequeue(name="dequeue_top")
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 421, in dequeue
    self._queue_ref, self._dtypes, name=name)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2603, in _queue_dequeue_v2
    timeout_ms=timeout_ms, name=name)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_1_bucket_by_sequence_length/bucket/top_queue' is closed and has insufficient elements (requested 1, current size 0)
   [[Node: bucket_by_sequence_length/bucket/dequeue_top = QueueDequeueV2[component_types=[DT_INT32, DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_STRING], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](bucket_by_sequence_length/bucket/top_queue)]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 111, in <module>
    plot_alignment(al[0], gs)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
    enqueue_callable()
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
    target_list_as_strings, status, None)
  File "/Users/me/anaconda3/envs/tacotron/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: a bytes-like object is required, not 'str'
   [[Node: PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_STRING, DT_FLOAT, DT_FLOAT], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/device:CPU:0"](input_producer/Gather)]]
(tacotron) MacBook-Air:tacotron me$ 

python prepro.py seems to be completed without errors.

Versions:
MacOS High Sierra: 10.13.6
Python: 3.6.6
Tensorflow: 1.4.0
numpy: 1.15.1
librosa: 0.6.2
tqdm: 4.25.0
matplotlib : 2.2.3
scipy: 1.1.0

I downloaded LJSpeech-1.1 and modified transcript.csv on data_load.py to metadata.csv because LJSpeech-1.1 don't have transcript.csv.

I used Anaconda to install python and pip for other libraries.

I want to feel how text to speech model works locally.

Thanks in advance.

[pandaGst]

Hello,kouohhashi.
I have a problem as same as yours. I run prepro.py completely and have no errors. but when I run train.py after prepro.py, I got this error.
Do you have any idea about this? Thank you very much!

[dwanderton]

+1

[dwanderton]

Seemingly solved when I use a requirements.txt as follows

NumPy >= 1.12.1
TensorFlow == 1.3
librosa
tqdm
matplotlib
scipy

[jooddang]

hmm. @dwanderton thanks for the comment but with tensorflow==1.3 I still get the same error message:

~/Documents/tacotron]$ python train.py 
Training Graph loaded
2018-12-26 14:40:18.411812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-26 14:40:18.411836: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-12-26 14:40:18.411842: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-26 14:40:18.411846: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
  0%|                                          | 0/409 [00:00<?, ?b/s]Bus error: 10

and my pip looks like this (under conda env)

Package                Version   
---------------------- ----------
absl-py                0.6.1     
astor                  0.7.1     
audioread              2.1.6     
bleach                 1.5.0     
certifi                2018.11.29
cycler                 0.10.0    
decorator              4.3.0     
gast                   0.2.0     
grpcio                 1.17.1    
h5py                   2.9.0     
html5lib               0.9999999 
joblib                 0.13.0    
Keras-Applications     1.0.6     
Keras-Preprocessing    1.0.5     
kiwisolver             1.0.1     
librosa                0.6.2     
llvmlite               0.26.0    
Markdown               3.0.1     
matplotlib             3.0.2     
numba                  0.41.0    
numpy                  1.15.4    
pip                    18.1      
protobuf               3.6.1     
pyparsing              2.3.0     
python-dateutil        2.7.5     
resampy                0.2.1     
scikit-learn           0.20.2    
scipy                  1.2.0     
setuptools             40.6.3    
six                    1.12.0    
tensorboard            1.12.1    
tensorflow             1.3.0     
tensorflow-tensorboard 0.1.8     
termcolor              1.1.0     
tqdm                   4.28.1    
Werkzeug               0.14.1    
wheel                  0.32.3   

[destiny311]

你们现在有解决这个问题吗

[gldstrrbt]

Commenting out a few lines seems to have done the trick.

In the file data_load.py, replace the following block:

     if hp.prepro:
        def _load_spectrograms(fpath):
            fname = os.path.basename(fpath)
            mel = "mels/{}".format(fname.replace("wav", "npy"))
            mag = "mags/{}".format(fname.replace("wav", "npy"))
            return fname, np.load(mel), np.load(mag)

        fname, mel, mag = tf.py_func(_load_spectrograms, [fpath], [tf.string, tf.float32, tf.float32])
    else:
        fname, mel, mag = tf.py_func(load_spectrograms, [fpath], [tf.string, tf.float32, tf.float32])  # (None, n_mels)

With the following:

    # if hp.prepro:
    #     def _load_spectrograms(fpath):
    #         fname = os.path.basename(fpath)
    #         mel = "mels/{}".format(fname.replace("wav", "npy"))
    #         mag = "mags/{}".format(fname.replace("wav", "npy"))
    #         return fname, np.load(mel), np.load(mag)

    #     fname, mel, mag = tf.py_func(_load_spectrograms, [fpath], [tf.string, tf.float32, tf.float32])
    # else:
    fname, mel, mag = tf.py_func(load_spectrograms, [fpath], [tf.string, tf.float32, tf.float32])  # (None, n_mels)