Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

floating point exception in tf.nn.max_pool3d when value of ksize is too large. #65120

Open
xl1043237213 opened this issue Apr 5, 2024 · 3 comments
Assignees
Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.15 For issues related to 2.15.x type:bug Bug

Comments

@xl1043237213
Copy link

xl1043237213 commented Apr 5, 2024

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

binary

TensorFlow version

tf 2.15

Custom code

No

OS platform and distribution

Linux Ubuntu 20.04

Mobile device

No response

Python version

3.10

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

floating point exception in tf.nn.max_pool3d when value of ksize is too large, which will cause a crash. This crash occurs in the CPU execution environment.

Standalone code to reproduce the issue

https://colab.research.google.com/drive/1S9D9zdhknu9ReoPahd0_kw2OEuRxLAvB?usp=sharing

Relevant log output

Floating point exception (core dumped)
@google-ml-butler google-ml-butler bot added the type:bug Bug label Apr 5, 2024
@tilakrayal tilakrayal added TF 2.15 For issues related to 2.15.x comp:ops OPs related issues labels Apr 8, 2024
@tilakrayal
Copy link
Contributor

@xl1043237213,
It's not being killed because of TF. You are literally allocating so much memory that the OS is killing the process. It seems like you're giving large negative value for multiple arguments to the function. so due to Integer overflow to buffer overflow or due to insufficient memory (RAM), code is crashing or the process getting killed and I was able to replicate the issue on Google colab. Could you please specify the usecase where you are trying with the large ksize.

input: A 5-D Tensor of the format specified by data_format.
ksize: An int or list of ints that has length 1, 3 or 5. The size of the window for each dimension of the input tensor.

Thank you!

@tilakrayal tilakrayal added the stat:awaiting response Status - Awaiting response from author label Apr 8, 2024
@xl1043237213
Copy link
Author

Thanks for your response. : )
I found the problem with a self-designed fuzzy testing tool, and have not yet found a relevant application scenario that requires setting ksize to such a large value. However, when the value of ksize is accessible to the user, packed in the model, it can cause the program to hang or even crash.

@google-ml-butler google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Apr 8, 2024
@tilakrayal
Copy link
Contributor

I was able to reproduce the issue on tensorflow v2.15 and v2.16. Kindly find the gist of it here.

@tilakrayal tilakrayal added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.15 For issues related to 2.15.x type:bug Bug
Projects
None yet
Development

No branches or pull requests

2 participants