Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem: corrupted size vs. prev_size when using python3.6 #10567

Closed
NewEricWang opened this issue Aug 16, 2018 · 6 comments
Closed

problem: corrupted size vs. prev_size when using python3.6 #10567

NewEricWang opened this issue Aug 16, 2018 · 6 comments

Comments

@NewEricWang
Copy link

Hi, I install tensorflow-1.9; then install pytorch=0.4 at python2 and python3 respectively.
I use python2 to run my code, no error is reported; but when using python3 to run it, an error is report after finishing run. The error like as follows:
Error in `/restools/tools/miniconda/miniconda3/bin/python': corrupted size vs. prev_size: 0x000055b2bcb5c690 ***

@zou3519
Copy link
Contributor

zou3519 commented Aug 20, 2018

Does installing pytorch 0.4.1 help at all?

@NewEricWang
Copy link
Author

@zou3519 , pytorch 0.4.1 has not help.

@NewEricWang
Copy link
Author

Hi, I found the solution of the error. Just export the following variable:
export LD_PRELOAD="/usr/lib/libtcmalloc_minimal.so.4"
But I don’t understand why it fixes the problem, and so sadly :(!

Furthermore, my other script will report error when using the setting.

@hyperfraise
Copy link
Collaborator

Same problem under python3.7, torch 1.0.1, and this doesn't fix it :/ Problem is espacially fierce (and not deterministic) when training groupnorms

@ngimel
Copy link
Collaborator

ngimel commented Feb 16, 2021

Closing due to age

@ngimel ngimel closed this as completed Feb 16, 2021
@NagabhushanSN95
Copy link

Hi, I found the solution of the error. Just export the following variable: export LD_PRELOAD="/usr/lib/libtcmalloc_minimal.so.4" But I don’t understand why it fixes the problem, and so sadly :(!

Furthermore, my other script will report error when using the setting.

Man! Thanks a ton for this. This worked!

I've tried so many things and couldn't fix this problem at all. One of my GPUs was effectively unusable due to this. You saved me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants