During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string

[goreng2]

Hi,
First, I use conda env
When I run python test_streaming_transducer.py, I encounter the below error

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
         [[{{node StatefulPartitionedCall/StatefulPartitionedCall_2/map/while/body/_549/map/while/streaming_transducer_beam_search/TensorArrayV2Write_2/TensorListSetItem/_188}}]]
  (1) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
         [[{{node StatefulPartitionedCall/StatefulPartitionedCall_2/map/while/body/_549/map/while/streaming_transducer_beam_search/TensorArrayV2Write_2/TensorListSetItem/_188}}]]
         [[StatefulPartitionedCall/StatefulPartitionedCall_1/map/while/body/_438/map/while/streaming_transducer_beam_search/while/body/_1485/map/while/streaming_transducer_beam_search/while/while/body/_2063/map/while/streaming_transducer_beam_search/while/while/while/body/_2487/map/while/streaming_transducer_beam_search/while/while/while/cond/pivot_f/_2813/_653]]
0 successful operations.
0 derived errors ignored. [Op:__inference__test_function_26746]

Function call stack:
_test_function -> _test_function

I thought It's problem about tensorflow version
Because My env include the below

tensorboard               2.3.0              pyh4dce500_0
tensorboard-plugin-wit    1.6.0                      py_0
tensorflow                2.2.0           gpu_py37h1a511ff_0
tensorflow-addons         0.11.2                   pypi_0    pypi
tensorflow-base           2.2.0           gpu_py37h8a81be8_0
tensorflow-datasets       3.2.1                    pypi_0    pypi
tensorflow-estimator      2.3.0                    pypi_0    pypi
tensorflow-gpu            2.3.1                    pypi_0    pypi
tensorflow-metadata       0.25.0                   pypi_0    pypi
warprnnt-tensorflow       0.1                      pypi_0    pypi

I focused tensorflow 2.2.0, So I update that using pip install -U tensorflow

Now, My env include

tensorboard               2.3.0              pyh4dce500_0
tensorboard-plugin-wit    1.6.0                      py_0
tensorflow                2.3.1                    pypi_0    pypi
tensorflow-addons         0.11.2                   pypi_0    pypi
tensorflow-datasets       3.2.1                    pypi_0    pypi
tensorflow-estimator      2.3.0                    pypi_0    pypi
tensorflow-gpu            2.3.1                    pypi_0    pypi
tensorflow-metadata       0.25.0                   pypi_0    pypi
warprnnt-tensorflow       0.1                      pypi_0    pypi

Well.. I don't know Why tensorflow-base uninstalled

Anyway, The same error occur again...

What can I do for running test code?

Thanks

[nglehuy]

@JeongwoonPark The error says that dtype=tf.string is not supported for copying between GPU and CPU. I haven't tested on GPU so can you try wrapping with tf.device("/CPU:0"): before name_scope in this function so that the beam search only runs on cpu.

[gandroz]

I tried it without success. I also tried to set the option --cpu when testing.
The input tensor looks like :

(<tf.Tensor: shape=(1,), dtype=string, numpy=array([b'filename.flac'], dtype=object)>,
 <tf.Tensor: shape=(1, 361120), dtype=float32, numpy=array([[...]], dtype=float32)>, 
 <tf.Tensor: shape=(1, 280), dtype=int32, numpy=array([[...]], dtype=int32)>)

and I was wondering about the need to add the filename of the input audio

[nglehuy]

@gandroz The file name is for storing to the tsv file, after testing, the .tsv file is generated and then you can read the .tsv to compare the results between groundtruth and other decoding methods and know where the corresponding audio file is stored.

[nglehuy]

@gandroz Can you send me the error when you use --cpu option?

[gandroz]

That's the same error

Traceback (most recent call last):
  File "examples/conformer/test_conformer.py", line 97, in <module>
    conformer_tester.run(test_dataset)
  File "/home/guillaume/src/desjardins/TensorFlowASR/tensorflow_asr/runners/base_runners.py", line 405, in run
    self._test_epoch()
  File "/home/guillaume/src/desjardins/TensorFlowASR/tensorflow_asr/runners/base_runners.py", line 417, in _test_epoch
    decoded = self._test_function(test_iter)
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 846, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 545, in call
    outputs = execute.execute(
  File "/home/guillaume/miniconda3/envs/stt/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
         [[{{node StatefulPartitionedCall/StatefulPartitionedCall/map/while/body/_2395/map/while/Identity_3/_1938}}]]
  (1) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
         [[{{node StatefulPartitionedCall/StatefulPartitionedCall/map/while/body/_2395/map/while/Identity_3/_1938}}]]
         [[StatefulPartitionedCall/StatefulPartitionedCall_1/map/while/body/_3023/map/while/map/while/loop_body_control/_9834/_2019]]
0 successful operations.
0 derived errors ignored. [Op:__inference__test_function_50389]

Function call stack:
_test_function -> _test_function

[gandroz]

The test runs if I explicitly tell to run on the CPU with tf.device("/CPU:0"): in function _test_function, of course no matter the option --cpu or --device used

Indeed, in the function iextract of the text_featurizer, the labels are computed on CPU as string data is not supported on GPU, but the beam search and greedy decoding are still on GPU. So wrapping the whole _test_function to work on CPU solves the issue.

The issue comes from the tf.map_fn. Even if the text_featurizer returns tensors hosted on CPU, the mapper redirect them to GPU apparently, even with the --cpu option. The second workaround proposed is to do something like

with tf.device("/CPU:0"):
    val = tf.map_fn(....)
return val

instead of just

return tf.map_fn(...)

in recognize and recognize_beam

I dont know if it has an impact for training though

[tund]

@JeongwoonPark The error says that dtype=tf.string is not supported for copying between GPU and CPU. I haven't tested on GPU so can you try wrapping with tf.device("/CPU:0"): before name_scope in this function so that the beam search only runs on cpu.

I was also faced with this issue when using tensorflow 2.2 (doesn't have fn_output_signatures, so need to use dtype=tf.string), and @usimarit 's suggestion (quoted) worked for me. Another way is to put with tf.device('/cpu:0') before every tf.map_fn(...., dtype=tf.string). However, I guess this is not really good in terms of the running time because of switching between GPU and CPU. May I know what you guys think?

[nglehuy]

@gandroz Training doesn't use any recognize function, so it doesn't slow down the training.
I'll find a way to fix this problem, in the meantime, you guys can try replace tf.map_fn with tf.while_loop and remove tf.device('/cpu:0') at every map_fn to see how it runs. I can do that but it'll take a while since I don't own any GPU.

[gandroz]

Training doesn't use any recognize function, so it doesn't slow down the training.

right, but we speak about testing here.

I'll try your suggestion, thanks

[nglehuy]

I updated the code using tf.while_loop, can you guys give it a try?