An Error occured when using maxpooling

[jingyi-zhang]

The error information is:

 **heatmap_output = max_pooling(input)**
  File "/home/zhangjingy/anaconda3/envs/image_pc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zhangjingy/project/image_pc/tools/../../MinkowskiEngine-master/MinkowskiEngine/MinkowskiPooling.py", line 524, in forward
    input.coords_man)
  File "/home/zhangjingy/project/image_pc/tools/../../MinkowskiEngine-master/MinkowskiEngine/MinkowskiPooling.py", line 84, in forward
    ctx.coords_man.CPPCoordsManager)
ValueError: Reduction size mismatch. out_nrows: 70309, num_unique_out_map: 1

The input of max_pooling is:

I initialized maxpooling like:
from MinkowskiEngine import MinkowskiMaxPooling as Max_Pooling max_pooling = Max_Pooling(kernel_size=3, stride=1, dimension=3)

[chrischoy]

Hmm. the maxpool unittests for both CPU and GPU pass successfully.

python -m tests.pool.

Could you provide a more context on the operation or some self-contained code?
Also, are you using the latest version of the Minkowski Engine?
pip3 install -U -I git+https://github.com/StanfordVL/MinkowskiEngine

[jingyi-zhang]

Thanks for your advice, but when I tried install latest version, another error occured:
(image_pc) zhangjingy@yons-desktop:~/project$ pip install -U -I git+https://github.com/StanfordVL/MinkowskiEngine
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting git+https://github.com/StanfordVL/MinkowskiEngine
Cloning https://github.com/StanfordVL/MinkowskiEngine to /tmp/pip-req-build-77nbc3qo
Running command git clone -q https://github.com/StanfordVL/MinkowskiEngine /tmp/pip-req-build-77nbc3qo
ERROR: Command errored out with exit status 1:
command: /home/zhangjingy/anaconda3/envs/image_pc/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-77nbc3qo/setup.py'"'"'; file='"'"'/tmp/pip-req-build-77nbc3qo/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-77nbc3qo/pip-egg-info
cwd: /tmp/pip-req-build-77nbc3qo/
Complete output (95 lines):
CXX src/region.cpp
CXX src/coordsmap.cpp
CXX src/pooling_global_max.cpp
CXX src/pooling_transpose.cpp
CXX src/convolution_transpose.cpp
CXX src/convolution.cpp
CXX src/coords_manager.cpp
CXX src/broadcast.cpp
CXX src/union.cpp
CXX src/pooling_avg.cpp
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/coords_manager.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/pooling_global_max.cpp:21:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/convolution_transpose.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/broadcast.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/pooling_transpose.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/convolution.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/union.hpp:28,
from src/union.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
In file included from src/coords_manager.hpp:46:0,
from src/common.hpp:34,
from src/pooling_avg.cpp:25:
src/gpu_memory_manager.hpp: In member function ‘void* minkowski::GPUMemoryManager::gpuMalloc(size_t)’:
src/gpu_memory_manager.hpp:111:51: error: ‘raw_alloc_with_stream’ is not a member of ‘c10::cuda::CUDACachingAllocator’
p_buffer = c10::cuda::CUDACachingAllocator::raw_alloc_with_stream(
^~~~~~~~~~~~~~~~~~~~~
Makefile:162: recipe for target 'objs/pooling_global_max.o' failed
make: *** [objs/pooling_global_max.o] Error 1
make: *** 正在等待未完成的任务....
Makefile:162: recipe for target 'objs/convolution_transpose.o' failed
make: *** [objs/convolution_transpose.o] Error 1
Makefile:162: recipe for target 'objs/broadcast.o' failed
make: *** [objs/broadcast.o] Error 1
Makefile:162: recipe for target 'objs/coords_manager.o' failed
make: *** [objs/coords_manager.o] Error 1
Makefile:162: recipe for target 'objs/pooling_transpose.o' failed
make: *** [objs/pooling_transpose.o] Error 1
Makefile:162: recipe for target 'objs/convolution.o' failed
make: *** [objs/convolution.o] Error 1
Makefile:162: recipe for target 'objs/union.o' failed
make: *** [objs/union.o] Error 1
Makefile:162: recipe for target 'objs/pooling_avg.o' failed
make: *** [objs/pooling_avg.o] Error 1

Using BLAS=openblas
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-77nbc3qo/setup.py", line 188, in <module>
    run_command(*compile_args)
  File "/tmp/pip-req-build-77nbc3qo/setup.py", line 90, in run_command
    subprocess.check_call(args)
  File "/home/zhangjingy/anaconda3/envs/image_pc/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('make', '-j8', 'PYTHON=/home/zhangjingy/anaconda3/envs/image_pc/bin/python3.7', 'BLAS=openblas')' returned non-zero exit status 2.
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
@chrischoy

[jingyi-zhang]

[chrischoy]

Ah the latest code requires pytorch 1.5

[jingyi-zhang]

I am sorry I ignored this requirement, but after I updated torch, it still failed.
information about:
/usr/local/cuda/include/cuda_runtime_api.h:9600:56: error: ‘cudaGraph_t’ was not declared in this scope
extern host cudaError_t CUDARTAPI cudaGraphDestroy(cudaGraph_t graph);
^~~~~~~~~~~
/usr/local/cuda/include/cuda_runtime_api.h:9600:56: note: suggested alternative: ‘cudaError_t’
extern host cudaError_t CUDARTAPI cudaGraphDestroy(cudaGraph_t graph);
^~~~~~~~~~~
cudaError_t
Makefile:162: recipe for target 'objs/quantization.o' failed
make: *** [objs/quantization.o] Error 1
Traceback (most recent call last):
File "setup.py", line 188, in
run_command(*compile_args)
File "setup.py", line 90, in run_command
subprocess.check_call(args)
File "/home/zhangjingy/anaconda3/envs/image_pc/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('make', '-j8', 'PYTHON=/home/zhangjingy/anaconda3/envs/image_pc/bin/python', 'BLAS=openblas')' returned non-zero exit status 2.

[chrischoy]

Seems to be related to #138.

[jingyi-zhang]

I am sorry to trouble you again, I changed “driver_types.h” to <driver_types.h>, but it didn't work. then I tried reinstall cudnn, it didn't work neither, I searched in torch forums, but I didn't get any result.

[jingyi-zhang]

Hi, Chrischoy, thanks for your selfless help, it's so important for me, fortunately, I installed minkowski 0.4.2 succeed by using docker, I want to push my images publicly, but it failed, if you think it is helpful, I will try to push again tomorrow!
@chrischoy

[jingyi-zhang]

I am sorry I ignored this requirement, but after I updated torch, it still failed.
information about:
/usr/local/cuda/include/cuda_runtime_api.h:9600:56: error: ‘cudaGraph_t’ was not declared in this scope
extern host cudaError_t CUDARTAPI cudaGraphDestroy(cudaGraph_t graph);
^~~~~~~~~~~
/usr/local/cuda/include/cuda_runtime_api.h:9600:56: note: suggested alternative: ‘cudaError_t’
extern host cudaError_t CUDARTAPI cudaGraphDestroy(cudaGraph_t graph);
^~~~~~~~~~~
cudaError_t
Makefile:162: recipe for target 'objs/quantization.o' failed
make: *** [objs/quantization.o] Error 1
Traceback (most recent call last):
File "setup.py", line 188, in
run_command(*compile_args)
File "setup.py", line 90, in run_command
subprocess.check_call(args)
File "/home/zhangjingy/anaconda3/envs/image_pc/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('make', '-j8', 'PYTHON=/home/zhangjingy/anaconda3/envs/image_pc/bin/python', 'BLAS=openblas')' returned non-zero exit status 2.

solved by reinstall cuda and cudnn~