Invalid free in binaries on error message

[jatentaki]

• OS: Xubuntu 16.04 4.13.0-32-generic
• PyTorch version: 0.3.0.post4
• How you installed PyTorch (conda, pip, source): pip
• Python version: Python 3.5.2
• CUDA/cuDNN version: running on CPU

To reproduce, clone this repo and execute python3 run_cnn.py. On my machine it results in the following stack trace.

I believe the script is self-explanatory, I was trying to maxpool a 3d convolved tensor :)

[zou3519]

@li-roy, can you take a look at this?

[li-roy]

@jatentaki I couldn't reproduce this. I get an invalid argument error. Were you using different numbers initially?

RuntimeError: invalid argument 2: pad should be smaller than half of kernel size, but got kT: 1 kW: 3, kH: 3, padT: 1, padW: 0, padH: 0 at /data/users/royboy/pytorch/torch/lib/THNN/generic/VolumetricDilatedMaxPooling.c:52

[jatentaki]

No, re-cloning the repo reproduces the bug for me. Also, I've just upgraded 3.0post4 to 3.1 and it also reproduces. Aren't you using a source-build master?

Also, if this runtime check is the go-to solution, it leads me to ask: if I have the RGB axis of an image, and I want to maxpool like

input = [R, G, B]
output = [max(0, R, G), max(R, G, B), max(G, B, 0)]

how should I go about it with this argument check? That was actually my intent with this code (along with some maxing along the W/H dimensions as well).

EDIT: Nvm the above, I confused it with a different layer. This snippet is definitely a bug on my side and should indeed be caught with invalid argument exception.

EDIT2: I am running the no-cuda wheel. Perhaps that makes the difference?

[soumith]

So, the invalid free is a bit of a side-effect of the wheel binaries that we have.
Your original issue is what @li-roy reported, i.e.

RuntimeError: invalid argument 2: pad should be smaller than half of kernel size, but got kT: 1 kW: 3, kH: 3, padT: 1, padW: 0, padH: 0 at /data/users/royboy/pytorch/torch/lib/THNN/generic/VolumetricDilatedMaxPooling.c:52

He couldn't reproduce it probably because of using conda binaries or source installs.

the rabbit-hole is deep, but the happens roughly because of this:

• we statically link stdc++ with binaries (because old OSes usually dont have a sufficiently recent stdc++ shipped)
• additionally because a bug in the libstdc++ shipped with RHEL6, we also statically link weak symbols (that includes destructors)
• the consequence of statically linking destructors of libstdc++ is a bit iffy. For certain internal data structures like std::string, each instance of libstdc++ symbols (my understanding / speculation) do their own book-keeping for construction/destruction, i.e. each string is sort of interned in a sense, internal to the library.

So what actually is happening is

• a std::string was passed from the THNN extension side over to _C land
• _C, after finishing using the string, calls the destructor on it
• but because _C didn't actually allocate the string, it's internal table doesn't have a reference to the pointer, so it sees it as an invalid free

In the next binary release, I'll figure out a solution for this.

Thanks a lot for the issue and the repro.