Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault (core dumped) in tf.raw_ops.FractionalMaxPoolGrad #66760

Open
LongZE666 opened this issue May 1, 2024 · 3 comments
Open
Assignees
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.16 type:bug Bug

Comments

@LongZE666
Copy link

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

tf 2.16.1

Custom code

Yes

OS platform and distribution

Ubuntu 20.04

Mobile device

No response

Python version

3.11

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

If row_pooling_sequence is a minimal negative number, tf.raw_ops.FractionalMaxPoolGrad encounters "Segmentation fault (core dumped)".

Standalone code to reproduce the issue

# FractionalMaxPoolGradOp

import tensorflow as tf

overlapping = True
orig_input = tf.constant(100, shape=[1,4,7,1], dtype=tf.float32)
orig_output = tf.constant(.453409232, shape=[1,7,13,1], dtype=tf.float32)
out_backprop = tf.constant(.453409232, shape=[1,7,13,1], dtype=tf.float32)
row_pooling_sequence = tf.constant(-1250999896764, shape=[5], dtype=tf.int64)
col_pooling_sequence = tf.constant(0, shape=[5], dtype=tf.int64)
tf.raw_ops.FractionalMaxPoolGrad(orig_input=orig_input, orig_output=orig_output, out_backprop=out_backprop, row_pooling_sequence=row_pooling_sequence, col_pooling_sequence=col_pooling_sequence, overlapping=overlapping)

Relevant log output

ASAN Report:
'''
AddressSanitizer:DEADLYSIGNAL
=================================================================
==870==ERROR: AddressSanitizer: SEGV on unknown address 0x412468dbc9b0 (pc 0x7fe87681d217 bp 0x7ffc4041d560 sp 0x7ffc4041c680 T0)
==870==The signal is caused by a READ memory access.
    #0 0x7fe87681d217 in tensorflow::FractionalMaxPoolGradOp<float>::Compute(tensorflow::OpKernelContext*) (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_cc.so.2+0x4139e217)
    #1 0x7fe89f4e714a in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2+0x1cdc14a)
    #2 0x7fe89f06dfe6 in tensorflow::(anonymous namespace)::SingleThreadedExecutorImpl::Run(tensorflow::Executor::Args const&) (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2+0x1862fe6)
    #3 0x7fe89ef4a306 in tensorflow::FunctionLibraryRuntimeImpl::RunSync(tensorflow::FunctionLibraryRuntime::Options, unsigned long, absl::lts_20230802::Span<tensorflow::Tensor const>, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2+0x173f306)
    #4 0x7fe89efa7f74 in tensorflow::ProcessFunctionLibraryRuntime::RunMultiDeviceSync(tensorflow::FunctionLibraryRuntime::Options const&, unsigned long, std::vector<std::variant<tensorflow::Tensor, tensorflow::TensorShape>, std::allocator<std::variant<tensorflow::Tensor, tensorflow::TensorShape> > >*, std::function<absl::lts_20230802::Status (tensorflow::ProcessFunctionLibraryRuntime::ComponentFunctionData const&, tensorflow::ProcessFunctionLibraryRuntime::InternalArgs*)>) const (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2+0x179cf74)
    #5 0x7fe89efb58b1 in tensorflow::ProcessFunctionLibraryRuntime::RunSync(tensorflow::FunctionLibraryRuntime::Options const&, unsigned long, absl::lts_20230802::Span<tensorflow::Tensor const>, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*) const (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_framework.so.2+0x17aa8b1)
    #6 0x7fe8670d72eb in tensorflow::KernelAndDeviceFunc::Run(tensorflow::ScopedStepContainer*, tensorflow::EagerKernelArgs const&, std::vector<std::variant<tensorflow::Tensor, tensorflow::TensorShape>, std::allocator<std::variant<tensorflow::Tensor, tensorflow::TensorShape> > >*, tsl::CancellationManager*, std::optional<tensorflow::EagerFunctionParams> const&, std::optional<tensorflow::ManagedStackTrace> const&, tsl::CoordinationServiceAgent*) (/tensorflow_src/venv/tensorflow-2.16.1-asan/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_cc.so.2+0x31c582eb)
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.
'''
@tilakrayal
Copy link
Contributor

@LongZE666,
Thank you for reporting the issue. I was able to reproduce the issue on tensorflow v2.15, v2.16 and tf-nightly. Kindly find the gist of it here and the screenshot of the check fail for the reference.

image

@tilakrayal tilakrayal added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label May 7, 2024
@tilakrayal
Copy link
Contributor

@LongZE666,
I tried to execute the mentioned code on the Tensorflow v2.16 in the alternative approach and it was failing with the error which was intended. Kindly find the gist of it here.

row_pooling_sequence--A Tensor of type int64. row pooling sequence, form pooling region with col_pooling_sequence

Thank you!

@tilakrayal tilakrayal added the stat:awaiting response Status - Awaiting response from author label Jun 5, 2024
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.16 type:bug Bug
Projects
None yet
Development

No branches or pull requests

2 participants