Segfault when running tensorflow.python.ops.gen_array_ops.upper_bound

[dmc1778]

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

No

Source

binary

Tensorflow Version

2.11.0

Custom Code

Yes

OS Platform and Distribution

Ubuntu 22.04

Mobile device

No response

Python version

3.9

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

A bug happened!

Standalone code to reproduce the issue

import tensorflow as tf
import numpy as np
from tensorflow.python.ops import gen_array_ops
try:
  try:
    with tf.device('/CPU'):
      arg_0_tensor = tf.random.uniform([2, 3], dtype=tf.float32)
      arg_0 = tf.identity(arg_0_tensor)
      arg_1_tensor = tf.random.uniform([2, 1, 0], dtype=tf.float32)
      arg_1 = tf.identity(arg_1_tensor)
      arg_2 = tf.int32
      arg_3 = None
      out = gen_array_ops.upper_bound(arg_0,arg_1,arg_2,arg_3,)
  except Exception as e:
    print("Error:"+str(e))
  try:
    with tf.device('/GPU:0'):
      arg_0 = tf.identity(arg_0_tensor)
      arg_0 = tf.cast(arg_0, tf.float32)
      arg_1 = tf.identity(arg_1_tensor)
      arg_1 = tf.cast(arg_1, tf.float32)
      arg_2 = tf.int32
      gen_array_ops.upper_bound(arg_0,arg_1,arg_2,arg_3,)
  except Exception as e:
    print("Error:"+str(e))
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

```shell
2023-01-22 11:31:45.460968: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-22 11:31:46.001164: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/
2023-01-22 11:31:46.001329: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/:/home/nimashiri/anaconda3/envs/cuda11.2/lib/
2023-01-22 11:31:46.001336: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-01-22 11:31:46.481782: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.486235: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.486347: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.486642: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-22 11:31:46.487105: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.487222: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.487314: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.846119: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.846261: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.846358: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-22 11:31:46.846441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4292 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
Segmentation fault

</details>

[tilakrayal]

@nimashiri,
I tried the mentioned code in an alternative approach where we make the shape of the argument shape{tf.random.uniform([2, 3, 0])} similar to arg_1_tensor and it was executed without any issues. Kindly find the gist of it here. Thank you!

[dmc1778]

@tilakrayal I am running fuzz testing on tensorflow, would you please run my reported test case? the test case is machine generated.

[tilakrayal]

@sachinprasadhs,
I was able to reproduce the issue on tensorflow v2.11 and tf-nightly where I was receiving the similar Segfault error with the arg_0_tensor = tf.random.uniform([2, 3] and arg_0_tensor = tf.random.uniform([2, 1, 0]. Kindly find the gist of it here.

[mihaimaruseac]

This is a vulnerability that needs fixing.

@tilakrayal Given that this is a vulnerability, don't be too eager to close it (by adding the label).

@dmc1778 please stop posting vulns on GitHub. Please consult SECURITY.md for how to properly and ethically disclose them.

[SuryanarayanaY]

Tested the code on Ubuntu VM and got segfault in both TF2.10 and TF2.12.0-dev20230201.

With TF 2.11: Segmentation fault (core dumped)

(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ python 59406.py
2023-02-08 13:58:18.869358: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-08 13:58:18.989823: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-02-08 13:58:19.734097: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.10/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-02-08 13:58:19.734203: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/suryanarayanay/miniconda3/envs/tf/lib/:/home/suryanarayanay/miniconda3/envs/tf2.10/lib/:/home/suryanarayanay/miniconda3/envs/tf2.11/lib/
2023-02-08 13:58:19.734219: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2.11.0
2023-02-08 13:58:24.631864: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-08 13:58:25.958273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-02-08 13:58:25.959882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
Segmentation fault (core dumped)
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ 

With TF 2.12.0-dev20230201: Segmentation fault (core dumped)

(tf) suryanarayanay@surya-ubuntu-22-04:~$ python 59406.py
2023-02-08 13:59:29.235787: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-02-08 13:59:29.276048: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay
2023-02-08 13:59:29.291495: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-08 13:59:30.137136: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.12.0-dev20230201
2023-02-08 13:59:36.697714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38201 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-02-08 13:59:36.699323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38201 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
Segmentation fault (core dumped)
(tf) suryanarayanay@surya-ubuntu-22-04:~$ 

[ccrusius]

Fixed in 915884f

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No