Segmentation fault when running gen_nn_ops.fractional_avg_pool

[dmc1778]

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

source

Tensorflow Version

2.10.0

Custom Code

Yes

OS Platform and Distribution

Ubuntu 22.04

Mobile device

No response

Python version

3.9

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

segfault happens with negative list elements.

Standalone code to reproduce the issue

import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import gen_nn_ops
try:
  arg_0_tensor = tf.random.uniform([5, 20, 30, 3], dtype=tf.float64)
  arg_0 = tf.identity(arg_0_tensor)
  arg_1_0 = 2
  arg_1_1 = -5.267949192431123
  arg_1_2 = -52.58578643762691
  arg_1_3 = 1
  arg_1 = [arg_1_0,arg_1_1,arg_1_2,arg_1_3,]
  arg_2 = True
  arg_3 = True
  deterministic = True
  seed = 87654321
  seed2 = 341261001
  out = gen_nn_ops.fractional_avg_pool(arg_0,arg_1,arg_2,arg_3,deterministic=deterministic,seed=seed,seed2=seed2,)
except Exception as e:
  print("Error:"+str(e))

import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import gen_nn_ops
try:
  arg_0_tensor = tf.random.uniform([1, 10, 10, 1], dtype=tf.float64)
  arg_0 = tf.identity(arg_0_tensor)
  arg_1_0 = True
  arg_1_1 = -0.35668935305391647
  arg_1_2 = -0.7209753581353426
  arg_1_3 = -87
  arg_1 = [arg_1_0,arg_1_1,arg_1_2,arg_1_3,]
  arg_2 = True
  arg_3 = True
  deterministic = True
  seed = 87654321
  seed2 = 341261001
  out = gen_nn_ops.fractional_avg_pool(arg_0,arg_1,arg_2,arg_3,deterministic=deterministic,seed=seed,seed2=seed2,)
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

```shell
2023-01-07 13:44:10.489552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.493914: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.494017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.494307: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-07 13:44:10.494924: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.495025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.495113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.840688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.840834: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.840928: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-01-07 13:44:10.841010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4263 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
Error:{{function_node __wrapped__FractionalAvgPool_device_/job:localhost/replica:0/task:0/device:CPU:0}} Fractional average pooling is not yet supported on the batch nor channel dimension. [Op:FractionalAvgPool]
Error:{{function_node __wrapped__FractionalAvgPool_device_/job:localhost/replica:0/task:0/device:CPU:0}} Both seed and seed2 should be 0 if deterministic is false. [Op:FractionalAvgPool]
Error:Expected bool for argument 'pseudo_random' not -69.0.
Error:Value for attr 'T' of uint32 is not in the list of allowed values: float, double, int32, int64
	; NodeDef: {{node FractionalAvgPool}}; Op<name=FractionalAvgPool; signature=value:T -> output:T, row_pooling_sequence:int64, col_pooling_sequence:int64; attr=pooling_ratio:list(float),min=4; attr=pseudo_random:bool,default=false; attr=overlapping:bool,default=false; attr=deterministic:bool,default=false; attr=seed:int,default=0; attr=seed2:int,default=0; attr=T:type,allowed=[DT_FLOAT, DT_DOUBLE, DT_INT32, DT_INT64]> [Op:FractionalAvgPool]
Segmentation fault

</details>

[tilakrayal]

@nimashiri,
I tried to execute the mentioned code and the issue was fixed in the latest stable v2.11 and nightly. Kindly find the gist of it here.

[dmc1778]

@nimashiri, I tried to execute the mentioned code and the issue was fixed in the latest stable v2.11 and nightly. Kindly find the gist of it here.

Thanks for the consideration.

[tilakrayal]

@nimashiri,
As it was the bug on tensorflow v2.10, it has been resolved on the latest stable tensorflow v2.11 where we can observe that Segfault error has not been raised. Thank you!

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[dmc1778]

Also this one:

import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import gen_nn_ops
try:
  arg_0_tensor = tf.random.uniform([5, 20, 30, 3], dtype=tf.float64)
  arg_0 = tf.identity(arg_0_tensor)
  arg_1_0 = 1
  arg_1_1 = -15.267949192431123
  arg_1_2 = 0.0
  arg_1_3 = -1
  arg_1 = [arg_1_0,arg_1_1,arg_1_2,arg_1_3,]
  arg_2 = True
  arg_3 = False
  deterministic = True
  seed = 87654319
  seed2 = 341261001
  out = gen_nn_ops.fractional_avg_pool(arg_0,arg_1,arg_2,arg_3,deterministic=deterministic,seed=seed,seed2=seed2,)
except Exception as e:
  print("Error:"+str(e))

[tilakrayal]

Also this one:

import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import gen_nn_ops
try:
  arg_0_tensor = tf.random.uniform([5, 20, 30, 3], dtype=tf.float64)
  arg_0 = tf.identity(arg_0_tensor)
  arg_1_0 = 1
  arg_1_1 = -15.267949192431123
  arg_1_2 = 0.0
  arg_1_3 = -1
  arg_1 = [arg_1_0,arg_1_1,arg_1_2,arg_1_3,]
  arg_2 = True
  arg_3 = False
  deterministic = True
  seed = 87654319
  seed2 = 341261001
  out = gen_nn_ops.fractional_avg_pool(arg_0,arg_1,arg_2,arg_3,deterministic=deterministic,seed=seed,seed2=seed2,)
except Exception as e:
  print("Error:"+str(e))

This issue was already tracking with #59198. And the initial issue for Segmentation fault was not able to reproduce in tf v2.11 as mentioned in the above comment.

Kindly find the gist of it here where we can observe that Segfault error has not been raised and executed with error pooling_ratio cannot be smaller than 1 which were due to seed = 87654321 and seed2 = 341261001. Thank you!

[mihaimaruseac]

Looks like an issue on cleanup side.

@learning-to-play