InvalidArgumentError when running map_fn on strings inside a tf.function #28007

ipod825 · 2019-04-20T21:45:58Z

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
TensorFlow installed from (source or binary):
TensorFlow version (use command below):

conda install tensorflow-gpu==2.0-alpha

Python version:
3.7.1
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
cudatoolkit-10.0.130-0
cudnn-7.3.1-cuda10.0_0
GPU model and memory:
GeForce RTX 2080 Ti

Describe the current behavior
Running the provided code on GPUs leads to error message tensorflow.python.framework.errors_impl.InvalidArgumentError: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
Without feeding the tensor to the convolution layer, summary.image would succeed.

Describe the expected behavior
Should run smoothly.

Code to reproduce the issue

import tensorflow as tf
from tensorflow.keras import layers

H, W, C = 10, 10, 3
imgs = tf.zeros([10, H, W, C])
ds = tf.data.Dataset.from_tensor_slices(imgs)
ds = ds.batch(2)
conv = layers.Conv2D(32, (4, 4), strides=(2, 2), padding='same')


@tf.function
def run(img, i):
    conv(img)
    tf.summary.image('img', img, i)


if __name__ == "__main__":
    train_summary_writer = tf.summary.create_file_writer('/tmp/testsummary')
    with train_summary_writer.as_default():
        for i, img in enumerate(ds):
            run(img, i)

Other info / logs

TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-04-20 14:44:30.818841: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1700000000 Hz
2019-04-20 14:44:30.819976: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55b6fa788f50 executing computa
tions on platform Host. Devices:
2019-04-20 14:44:30.820029: I tensorflow/compiler/xla/service/service.cc:169]   StreamExecutor device (0): <undefined>, <u
ndefined>
2019-04-20 14:44:30.825689: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic li
brary libcuda.so.1
2019-04-20 14:44:31.062487: I tensorflow/compiler/xla/service/service.cc:162] XLA service 0x55b6fc634120 executing computa
tions on platform CUDA. Devices:
2019-04-20 14:44:31.062554: I tensorflow/compiler/xla/service/service.cc:169]   StreamExecutor device (0): GeForce RTX 208
0 Ti, Compute Capability 7.5
2019-04-20 14:44:31.063894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1467] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:19:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-04-20 14:44:31.063942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1546] Adding visible gpu devices: 0
2019-04-20 14:44:31.064034: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic li
brary libcudart.so.10.0
2019-04-20 14:44:31.067082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1015] Device interconnect StreamExecutor wi
th strength 1 edge matrix:
2019-04-20 14:44:31.067114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021]      0
2019-04-20 14:44:31.067130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1034] 0:   N
2019-04-20 14:44:31.068283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1149] Created TensorFlow device (/job:local
host/replica:0/task:0/device:GPU:0 with 10284 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id
: 0000:19:00.0, compute capability: 7.5)
2019-04-20 14:44:33.628228: W tensorflow/core/common_runtime/base_collective_executor.cc:214] BaseCollectiveExecutor::Star
tAbort Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
         [[{{node img_1/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_54}}]]
         [[img_1/encode_each_image/while/loop_body_control/_19/_33]]
2019-04-20 14:44:33.628374: W tensorflow/core/common_runtime/base_collective_executor.cc:214] BaseCollectiveExecutor::Star
tAbort Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
         [[{{node img_1/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_54}}]]
2019-04-20 14:44:33.628468: E tensorflow/core/common_runtime/process_function_library_runtime.cc:764] Component function e
xecution failed: Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
         [[{{node img_1/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_54}}]]
         [[img_1/encode_each_image/while/loop_body_control/_19/_33]]
2019-04-20 14:44:33.628456: E tensorflow/core/common_runtime/process_function_library_runtime.cc:764] Component function e
xecution failed: Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
         [[{{node img_1/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_54}}]]
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    run(img, i)
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/def_function.
py", line 438, in __call__
    return self._stateless_fn(*args, **kwds)
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/function.py",
 line 1288, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/function.py",
 line 574, in _filtered_call
    (t for t in nest.flatten((args, kwargs))
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/function.py",
 line 627, in _call_flat
    outputs = self._inference_function.call(ctx, args)
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/function.py",
 line 415, in call
    ctx=ctx)
  File "/home/swang150/.pyenv/versions/miniconda3-latest/lib/python3.7/site-packages/tensorflow/python/eager/execute.py",
line 66, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: During Variant Host->Device Copy: non-DMA-copy attempted of
tensor type: string
         [[{{node img_1/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_54}}]]
         [[img_1/encode_each_image/while/loop_body_control/_19/_33]] [Op:__inference_run_343]

The text was updated successfully, but these errors were encountered:

jvishnuvardhan · 2019-04-22T22:45:41Z

@ipod825 I ran the code in Google colab without any error. Could you try with Google colab and see whether issue persists there. If it doesn't persist in Google colab, then try to upgrade your TF2.0 and run the code again. Thanks!

!pip install tensorflow==2.0.0-alpha0
import tensorflow as tf
from tensorflow.keras import layers

H, W, C = 10, 10, 3
imgs = tf.zeros([10, H, W, C])
ds = tf.data.Dataset.from_tensor_slices(imgs)
ds = ds.batch(2)
conv = layers.Conv2D(32, (4, 4), strides=(2, 2), padding='same')


@tf.function
def run(img, i):
    conv(img)
    tf.summary.image('img', img, i)


if __name__ == "__main__":
    train_summary_writer = tf.summary.create_file_writer('/tmp/testsummary')
    with train_summary_writer.as_default():
        for i, img in enumerate(ds):
            run(img, i)
            print(i)

ipod825 · 2019-04-22T23:13:33Z

You need to run it on GPU.

!pip install tensorflow-gpu==2.0.0-alpha0
import tensorflow as tf
from tensorflow.keras import layers

H, W, C = 10, 10, 3
imgs = tf.zeros([10, H, W, C])
ds = tf.data.Dataset.from_tensor_slices(imgs)
ds = ds.batch(2)
conv = layers.Conv2D(32, (4, 4), strides=(2, 2), padding='same')


@tf.function
def run(img, i):
    conv(img)
    tf.summary.image('img', img, i)


if __name__ == "__main__":
    train_summary_writer = tf.summary.create_file_writer('/tmp/testsummary')
    with tf.device('/gpu:0'):
      with train_summary_writer.as_default():
          for i, img in enumerate(ds):
              run(img, i)
              print(i)

faruba · 2019-07-01T01:33:47Z

sorry about my poor English. I have the same problem.But I found a solution.
I'm using Nvidia 2080Ti , tf-nightly-gpu-2.0-preview. python3.7.3 ,ubuntu 19.04
When I used tf.summary.image("gen", generated_images, max_outputs=25, step=0), I got error :During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string.
If I wrote like this:

with tf.device("cpu:0"): <<-- add this line
   with log["writer"].as_default():
     tf.summary.image("gen", generated_images, max_outputs=25, step=0)

everything is fine.

jvishnuvardhan · 2019-08-13T22:51:14Z

Automatically closing this out since I understand it to be resolved, but please let me know if I'm mistaken.Thanks!

tensorflow-bot · 2019-08-13T22:51:16Z

Are you satisfied with the resolution of your issue?
Yes
No

ipod825 · 2019-08-13T23:14:54Z

Hi @jvishnuvardhan, the snippet I wrote still failed with the same error message in colab (using gpu accelerator). Even 2.0-beta version failed too. Could you please describe how you make it work?

jvishnuvardhan · 2019-08-15T22:42:44Z

@ipod825 As mentioned by @faruba I changed a line in your code from with tf.device('/gpu:0'): to with tf.device('/cpu:0'):. It works without any error. Please check the gist here. Thanks!

ipod825 · 2019-08-15T23:37:17Z

Well, it seems to be just a workaround to me. The main issue here is that the summary operation raises error when running on GPU. Forcing the operation to run on CPU doesn't really solve the problem but just ignores the problem. I don't know how summary operation works, probably even if running under GPU, it would still copy tensor back to CPU memory (which then would be similar to explicitly asking it to run on CPU). Even if this is the case (if not, we lose some efficiency), from an API point of view, I don't think this issue is solved as someone might encounter the same problem and don't know why it happen and how to solve it without bumping into this thread.

loffermann · 2019-08-19T11:43:32Z

@ipod825 I have the same problem (did try tf2.0 alphas and betas) and agree that assigning the summary op to /cpu:0 is only a workaround. Moreover, the fix is not working for me if I build from the r2.0 branch from source.
It would be nice if this issue would be reopened, so the problem can be solved.

I took a look at github: the map_fn in line 75 is causing the issues.

jvishnuvardhan · 2019-08-19T16:31:29Z

@ipod825 @loffermann Sorry for closing. Reopened. Thanks!

ahmadyan · 2019-09-18T17:38:21Z

I ran to a similar issue, when I tried to save images to tf.summary.image using MirroredStrategy. Oddly enough, when debugging using tf.config.experimental_run_functions_eagerly(True) this error does not occur.

(0) Invalid argument: 2 root error(s) found.
(0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
(1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
[[{{node training-image/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_62}}]]
[[Func/training-image/encode_each_image/while/body/_1/input/_50/_60]]
(1) Invalid argument: 2 root error(s) found.
(0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
(1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
[[{{node training-image/encode_each_image/while/body/_1/TensorArrayV2Write/TensorListSetItem/_62}}]]
0 successful operations.
0 derived errors ignored.
Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [Op:__inference_train_step_16043]

Function call stack:
train_step -> train_step

ageron · 2019-10-10T12:26:34Z

I also ran into this issue. Here's a fairly minimal piece of code that reproduces it:

import tensorflow.compat.v2 as tf
tf.enable_v2_behavior()

def decode_png(data):
  return tf.image.decode_png(data)

@tf.function  # <= No exception if you comment this line out
def decode_all(images):
  return tf.map_fn(decode_png, images, dtype=tf.uint8)

img = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\xdac\xfc\xcf\xf0\xbf\x1e\x00\x06\x83\x02\x7f\x94\xad\xd0\xeb\x00\x00\x00\x00IEND\xaeB`\x82'
images = tf.constant([img, img])
decode_all(images)

and here's the full stackstrace:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-72-a59f4c54298a> in <module>()
     11 img = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\xdac\xfc\xcf\xf0\xbf\x1e\x00\x06\x83\x02\x7f\x94\xad\xd0\xeb\x00\x00\x00\x00IEND\xaeB`\x82'
     12 images = tf.constant([img, img])
---> 13 decode_all(images)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    465               *args, **kwds)
    466       # If we did not create any variables the trace we have is good enough.
--> 467       return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
    468 
    469     def fn_with_cond(*inner_args, **inner_kwds):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py in _filtered_call(self, args, kwargs)
   1139          if isinstance(t, (ops.Tensor,
   1140                            resource_variable_ops.BaseResourceVariable))),
-> 1141         self.captured_inputs)
   1142 
   1143   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1222     if executing_eagerly:
   1223       flat_outputs = forward_function.call(
-> 1224           ctx, args, cancellation_manager=cancellation_manager)
   1225     else:
   1226       gradient_name = self._delayed_rewrite_functions.register()

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    509               inputs=args,
    510               attrs=("executor_type", executor_type, "config_proto", config),
--> 511               ctx=ctx)
    512         else:
    513           outputs = execute.execute_with_cancellation(

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     65     else:
     66       message = e.message
---> 67     six.raise_from(core._status_to_exception(e.code, message), None)
     68   except TypeError as e:
     69     keras_symbolic_tensors = [

/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node map/TensorArrayUnstack/TensorListFromTensor/_12}}]]
  (1) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node map/TensorArrayUnstack/TensorListFromTensor/_12}}]]
	 [[Func/map/while/body/_1/input/_43/_24]]
0 successful operations.
0 derived errors ignored. [Op:__inference_decode_all_20554]

Function call stack:
decode_all -> decode_all

I ran this on Colab with a GPU Runtime, using TF 1.15.0rc3. It will probably bomb as well on TF 2.0.0 but I haven't tried.

rharish101 · 2019-10-15T03:57:00Z

A modification of the earlier gist posted by @jvishnuvardhan causes the error to occur again. Here it is.

I am using both the GPU and CPU inside the decorated function because there might be a computationally expensive part that I have to run on the GPU. The CPU is used to run the summary ops as a workaround to this bug. However, the error still occurs. The only way to get around this is to run the entire function on the CPU when calling it, as @jvishnuvardhan 's gist, which is not ideal when I want to train a network.

dimitree54 · 2020-04-17T10:24:23Z

@Lannister-Xiaolin, I have not tried. I run program locally, without wrapping map_fn in keras.layer (as in your example).

TIGERCHANG123 · 2020-04-19T23:33:50Z

Hi, sorry for my poor English. I solved this problem by log the dataset images outside the @tf.function.

donbonjenbi · 2020-04-20T02:40:57Z

@TIGERCHANG123 - interesting! can you please share a code example?

schdief06 · 2020-04-20T08:47:09Z

I'm having the same issue using map_fn to decode a image list. Even when using my cpu.

def decode_image_fn(x):
  t = tf.image.decode_image(x, 3)
  t.set_shape((224, 224, 3))
  return t

frames = tf.map_fn(decode_image_fn, frames_raw, dtype=tf.uint8)

elpapi42 · 2020-04-22T21:34:17Z

Im having this issue too with TF2.2rc3

burhr2 · 2020-04-27T13:55:35Z

I am also facing this issue on GPU (no error on CPU) when using to map_fn for string tensor (with float tensor all right):

def process_string(sample):
    # here i want to write something to file, but even with identity
    # function I get "non-DMA-copy attempted..." error
    return sample

@tf.function
def f(self, y_true, y_pred):
    string_tensor = y_true["path"]  # Tensor("data_batch_19:0", shape=(2,), dtype=string)
    tf.map_fn(process_string, string_tensor)

I do not know why, but for me manually placing map_fn on cpu AND making mock return from tf.function helped:

@tf.function
def f(self, y_true, y_pred):
    string_tensor = y_true["path"]  # Tensor("data_batch_19:0", shape=(2,), dtype=string)
    with tf.device("/cpu:0"):
        tf.map_fn(process_string, string_tensor)
    return y_pred  # do not work for me without return something

I am using tf2.0.1

Hi thanks this trick works for my case also

saikumarchalla · 2020-07-02T04:59:25Z

@ipod825
Is this still an issue?
I was able to run the code without any issues with TF v2.2, please find the gist of it here.Thanks!

rharish101 · 2020-07-02T07:19:01Z

@saikumarchalla Yes, here's a Colab notebook.

tensorflow/tensorflow#28007

nikitamaia · 2020-08-06T18:36:52Z

@rharish101 @saikumarchalla I think this has been fixed. I just ran the colab from @rharish101 with tf nightly, and am not seeing an error. Please confirm.

rharish101 · 2020-08-07T07:01:16Z

@nikitamaia Yes, this is fixed on TensorFlow 2.3.0. I tested this on the colab and also on my local machine (running TF 2.3.0 on Arch Linux).

nikitamaia · 2020-08-07T14:34:44Z

Closing this issue now since the bug has been fixed.

google-ml-butler · 2020-08-07T14:34:46Z

Are you satisfied with the resolution of your issue?
Yes
No

hosford42 · 2021-02-22T17:14:56Z

I am getting a similar error on TF 2.4.1.

import tensorflow as tf


def _get_char_flags(char_tensor):
    char = char_tensor.numpy().decode()
    return char.isalpha(), char.isspace()


@tf.function  # <--------------- Runs fine with this line commented out.
def tf_split_text(text):
    tf.assert_rank(text, 0)
    tf.debugging.assert_type(text, tf.string)

    chars = tf.strings.unicode_split(text, input_encoding='UTF-8')
    is_alpha, is_space = tf.map_fn(lambda char: tf.py_function(_get_char_flags, char, Tout=[tf.bool, tf.bool]),
                                   (chars,), dtype=tf.bool, parallel_iterations=True,
                                   fn_output_signature=[tf.bool, tf.bool])

    is_alpha = tf.concat([is_alpha, [False]], axis=0)
    is_space = tf.concat([is_space, [True]], axis=0)

    is_special = ~(is_alpha | is_space)
    is_non_alpha = ~is_alpha
    is_non_space = ~is_space

    was_special = tf.concat([[False], is_special[:-1]], axis=0)
    was_non_alpha = tf.concat([[True], is_non_alpha[:-1]], axis=0)
    was_non_space = tf.concat([[False], is_non_space[:-1]], axis=0)

    any_to_special = is_special
    non_alpha_to_non_space = was_non_alpha & is_non_space
    token_start_flags = any_to_special | non_alpha_to_non_space
    token_start_indices = tf.where(token_start_flags)[:, 0]

    special_to_any = was_special
    non_space_to_non_alpha = was_non_space & is_non_alpha
    token_end_flags = special_to_any | non_space_to_non_alpha
    token_end_indices = tf.where(token_end_flags)[:, 0]

    tf.debugging.assert_equal(tf.size(token_start_indices), tf.size(token_end_indices))

    preceding_space = tf.concat([[False], is_space[:-1]], axis=0)

    tokens = tf.strings.substr(text, token_start_indices, token_end_indices - token_start_indices, unit='UTF8_CHAR')
    has_preceding_space = tf.gather(preceding_space, token_start_indices)

    tf.assert_rank(has_preceding_space, 1)
    tf.assert_rank(tokens, 1)
    tf.assert_equal(tf.reduce_sum(tf.map_fn(tf.strings.length, tokens, fn_output_signature=tf.int32)) +
                    tf.reduce_sum(tf.cast(has_preceding_space, tf.int32)),
                    tf.strings.length(text))
    return has_preceding_space, tokens


text = tf.constant('hi there', tf.string)
preceding_spaces, words = tf_split_text(text)
print(words)

Traceback (most recent call last):
  File "/home/hosford42/PycharmProjects/ImageParser/error.py", line 56, in <module>
    preceding_spaces, words = tf_split_text(text)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 894, in _call
    return self._concrete_stateful_fn._call_flat(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node map_1/TensorArrayUnstack/TensorListFromTensor/_96}}]]
	 [[map_1/while/loop_body_control/_61/_107]]
  (1) Invalid argument:  2 root error(s) found.
  (0) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
  (1) Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string
0 successful operations.
0 derived errors ignored.
	 [[{{node map_1/TensorArrayUnstack/TensorListFromTensor/_96}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_tf_split_text_265]

Function call stack:
tf_split_text -> tf_split_text

nikitamaia · 2021-02-22T17:26:19Z

Hi @hosford42, I just ran your code sample in Colab and I'm not seeing any errors. Please let me know if I'm missing something and open a new issue for further debugging.

hosford42 · 2021-02-22T18:48:15Z

@nikitamaia, I'm not using Colab, so perhaps the environment is the issue. I'm running Python 3.8.5 on an Ubuntu machine. I can provide more details if necessary.

Here is the full output when I start python and import tensorflow to show the version number:

Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.13.0
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
import tensorflow as tf
2021-02-22 12:46:55.445604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Invalid MIT-MAGIC-COOKIE-1 keyIn[3]: tf.__version__
Out[3]: '2.4.1'

hosford42 · 2021-02-22T18:50:54Z

And here is the logging output when I start a session, for GPU-specific info if that's needed:

2021-02-22 12:49:32.393693: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-22 12:49:32.395363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-22 12:49:32.441021: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.441279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 104.43GiB/s
2021-02-22 12:49:32.441313: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-22 12:49:32.444150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-22 12:49:32.444248: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-22 12:49:32.445154: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-22 12:49:32.445415: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-22 12:49:32.447089: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-22 12:49:32.448029: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-22 12:49:32.448393: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-22 12:49:32.448639: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.449058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.449257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-22 12:49:32.450007: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-22 12:49:32.450607: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-22 12:49:32.450814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.451052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 Ti computeCapability: 6.1
coreClock: 1.62GHz coreCount: 6 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 104.43GiB/s
2021-02-22 12:49:32.451087: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-22 12:49:32.451164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-22 12:49:32.451193: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-22 12:49:32.451221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-22 12:49:32.451248: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-22 12:49:32.451275: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-22 12:49:32.451302: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-22 12:49:32.451329: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-22 12:49:32.451431: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.451703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.451881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-22 12:49:32.451925: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-22 12:49:32.992073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-22 12:49:32.992131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2021-02-22 12:49:32.992141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2021-02-22 12:49:32.992400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.992681: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.992895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-22 12:49:32.993079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 142 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)

nikitamaia · 2021-02-22T19:13:40Z

Actually, I just ran the Colab gist that I shared in my earlier post, but this time with a GPU runtime. I'm now seeing the same error message that you reported. So seems to be a GPU related issue. Can you open a new bug with all of this information? Thanks!

achandraa self-assigned this Apr 22, 2019

achandraa added comp:ops OPs related issues type:bug Bug 2.0.0-alpha0 labels Apr 22, 2019

achandraa assigned jvishnuvardhan and unassigned achandraa Apr 22, 2019

jvishnuvardhan added the stat:awaiting response Status - Awaiting response from author label Apr 22, 2019

jvishnuvardhan assigned nfelt and unassigned jvishnuvardhan Apr 22, 2019

jvishnuvardhan added stat:awaiting tensorflower Status - Awaiting response from tensorflower and removed stat:awaiting response Status - Awaiting response from author labels Apr 22, 2019

jvishnuvardhan closed this as completed Aug 13, 2019

jvishnuvardhan reopened this Aug 19, 2019

goldiegadde added TF 2.0 Issues relating to TensorFlow 2.0 and removed TF 2.0.0-alpha0 labels Sep 12, 2019

goldiegadde added comp:core issues related to core part of tensorflow and removed comp:ops OPs related issues labels Oct 11, 2019

burhr2 mentioned this issue Apr 26, 2020

Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string affinelayer/pix2pix-tensorflow#188

Closed

gowthamkpr mentioned this issue May 26, 2020

serving model with base64 as input failed with gpu but ok with cpu tensorflow/serving#1626

Closed

saikumarchalla self-assigned this Jun 29, 2020

saikumarchalla removed their assignment Jul 11, 2020

bojiang mentioned this issue Jul 23, 2020

[FIX] tf fashion mnist example bentoml/gallery#39

Merged

BlueFisher added a commit to BlueFisher/Advanced-Soft-Actor-Critic that referenced this issue Jul 28, 2020

fix tf issue

9dfe305

tensorflow/tensorflow#28007

nikitamaia added the stat:awaiting response Status - Awaiting response from author label Aug 6, 2020

nikitamaia closed this as completed Aug 7, 2020

geetachavan1 added this to To do in TensorFlow 2.4.0 via automation Aug 27, 2020

geetachavan1 moved this from To do to Done in TensorFlow 2.4.0 Aug 27, 2020

frederick0329 mentioned this issue Aug 28, 2020

XAI notebooks ported to TF2 GoogleCloudPlatform/ai-platform-samples#195

Merged

hosford42 mentioned this issue Feb 22, 2021

Invalid argument: During Variant Host->Device Copy: non-DMA-copy attempted of tensor type: string #47325

Closed

UsharaniPagadala mentioned this issue Jun 30, 2021

Iterating through DistributedDataset created from dataset with raggedTensor raises InvalidArgumentError #50507

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

InvalidArgumentError when running map_fn on strings inside a tf.function #28007

InvalidArgumentError when running map_fn on strings inside a tf.function #28007

ipod825 commented Apr 20, 2019

jvishnuvardhan commented Apr 22, 2019

ipod825 commented Apr 22, 2019

faruba commented Jul 1, 2019 •

edited

jvishnuvardhan commented Aug 13, 2019

tensorflow-bot bot commented Aug 13, 2019

ipod825 commented Aug 13, 2019

jvishnuvardhan commented Aug 15, 2019

ipod825 commented Aug 15, 2019

loffermann commented Aug 19, 2019

jvishnuvardhan commented Aug 19, 2019

ahmadyan commented Sep 18, 2019 •

edited

ageron commented Oct 10, 2019 •

edited

rharish101 commented Oct 15, 2019 •

edited

dimitree54 commented Apr 17, 2020

TIGERCHANG123 commented Apr 19, 2020

donbonjenbi commented Apr 20, 2020

schdief06 commented Apr 20, 2020 •

edited

elpapi42 commented Apr 22, 2020

burhr2 commented Apr 27, 2020

saikumarchalla commented Jul 2, 2020

rharish101 commented Jul 2, 2020

nikitamaia commented Aug 6, 2020

rharish101 commented Aug 7, 2020

nikitamaia commented Aug 7, 2020

google-ml-butler bot commented Aug 7, 2020

hosford42 commented Feb 22, 2021

nikitamaia commented Feb 22, 2021

hosford42 commented Feb 22, 2021

hosford42 commented Feb 22, 2021

nikitamaia commented Feb 22, 2021

InvalidArgumentError when running map_fn on strings inside a tf.function #28007

InvalidArgumentError when running map_fn on strings inside a tf.function #28007

Comments

ipod825 commented Apr 20, 2019

jvishnuvardhan commented Apr 22, 2019

ipod825 commented Apr 22, 2019

faruba commented Jul 1, 2019 • edited

jvishnuvardhan commented Aug 13, 2019

tensorflow-bot bot commented Aug 13, 2019

ipod825 commented Aug 13, 2019

jvishnuvardhan commented Aug 15, 2019

ipod825 commented Aug 15, 2019

loffermann commented Aug 19, 2019

jvishnuvardhan commented Aug 19, 2019

ahmadyan commented Sep 18, 2019 • edited

ageron commented Oct 10, 2019 • edited

rharish101 commented Oct 15, 2019 • edited

dimitree54 commented Apr 17, 2020

TIGERCHANG123 commented Apr 19, 2020

donbonjenbi commented Apr 20, 2020

schdief06 commented Apr 20, 2020 • edited

elpapi42 commented Apr 22, 2020

burhr2 commented Apr 27, 2020

saikumarchalla commented Jul 2, 2020

rharish101 commented Jul 2, 2020

nikitamaia commented Aug 6, 2020

rharish101 commented Aug 7, 2020

nikitamaia commented Aug 7, 2020

google-ml-butler bot commented Aug 7, 2020

hosford42 commented Feb 22, 2021

nikitamaia commented Feb 22, 2021

hosford42 commented Feb 22, 2021

hosford42 commented Feb 22, 2021

nikitamaia commented Feb 22, 2021

faruba commented Jul 1, 2019 •

edited

ahmadyan commented Sep 18, 2019 •

edited

ageron commented Oct 10, 2019 •

edited

rharish101 commented Oct 15, 2019 •

edited

schdief06 commented Apr 20, 2020 •

edited