Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

centerpoint训练报错 ERROR - voxelize builded fail! PointPillars模型导出报错 ERROR - iou3d_nms_cuda builded fail! #118

Closed
yuwen37 opened this issue Oct 26, 2022 · 12 comments
Closed

centerpoint训练报错 ERROR - voxelize builded fail! PointPillars模型导出报错 ERROR - iou3d_nms_cuda builded fail! #118

yuwen37 opened this issue Oct 26, 2022 · 12 comments
Assignees
@will-jl944

Comments

@yuwen37
Copy link

yuwen37 commented Oct 26, 2022

我在用kitti数据训练时遇到了如下报错：

W1026 10:32:13.016722 15452 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 10.2, Runtime API Version: 10.2
W1026 10:32:13.016722 15452 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
2022-10-26 10:32:14,353 -     INFO - Finish CenterHead Initialization
2022-10-26 10:32:14,448 -     INFO - There are 191/191 variables loaded into CenterPoint.
2022-10-26 10:32:26,213 -  WARNING - No custom op voxelize found, try JIT build
Compiling user custom op, it will cost a few seconds.....
'D:\Program' �����ڲ����ⲿ���Ҳ���ǿ����еij���
���������ļ���
2022-10-26 10:32:26,239 -    ERROR - voxelize builded fail!
Traceback (most recent call last):
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 130, in _load_module
    self.module = importlib.import_module(self.modulename)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'voxelize'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1150, in run_cmd
    return subprocess.check_call(command, shell=True, stdout=DEVNULL)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cd /d C:\Users\decocloudadmin\.cache\paddle_extensions && D:\Program Files\Anaconda3\envs\paddle\python.exe voxelize_setup.py build' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/lyw/2022-8-12Detection/Paddle3D-develop/tools/train.py", line 189, in <module>
    main(args)
  File "E:/lyw/2022-8-12Detection/Paddle3D-develop/tools/train.py", line 184, in main
    trainer.train()
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\apis\trainer.py", line 219, in train
    self.cur_iter)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\apis\pipeline.py", line 27, in training_step
    outputs = model(sample)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\centerpoint\centerpoint.py", line 111, in forward
    x = self.extract_feat(data)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\centerpoint\centerpoint.py", line 88, in extract_feat
    data['points'])
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\voxelizers\voxelize.py", line 67, in forward
    point, max_num_voxels, bs_idx)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\voxelizers\voxelize.py", line 40, in single_forward
    voxels, coors, num_points_per_voxel, voxels_num = voxelize.hard_voxelize(
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 148, in __getattr__
    module = self._load_module()
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 134, in _load_module
    self.module = self.jit_build()
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 122, in jit_build
    name=self.modulename, sources=sources, **args)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\cpp_extension.py", line 857, in load
    _jit_compile(file_path, verbose)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1112, in _jit_compile
    run_cmd(compile_cmd, verbose)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1154, in run_cmd
    compile, error))
RuntimeError: Failed to run command: <built-in function compile>, errors: Command 'cd /d C:\Users\decocloudadmin\.cache\paddle_extensions && D:\Program Files\Anaconda3\envs\paddle\python.exe voxelize_setup.py build' returned non-zero exit status 1.

Process finished with exit code 1

不是很理解custom op voxelize是什么，不知道如何解决？

目前采用单卡训练，我的运行命令是：
python tools/train.py --config configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml --save_dir ./output_kitti --num_workers 0 --save_interval 5 --model pretrained_model/centerpoint/model.pdparams
@yuwen37
Copy link
Author

yuwen37 commented Oct 26, 2022

切换使用PointPillars模型时，训练没有报错，但是在export模型时，出现了一个很类似的错误

"D:\Program Files\Anaconda3\envs\paddle\python.exe" E:/lyw/2022-8-12Detection/Paddle3D-develop/tools/export.py --config configs/pointpillars/pointpillars_xyres16_kitti_car.yml --model outputs/pointpillars/iter_19800/model.pdparams --save_dir exported_model/PointPillars 
W1026 15:28:20.512589 14632 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 10.2, Runtime API Version: 10.2
W1026 15:28:20.517588 14632 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
2022-10-26 15:28:21,892 -     INFO - There are 106/106 variables loaded into PointPillars.
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmpi4hjhxyt.py:21
The behavior of expression A / B has been unified with elementwise_div(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_div(X, Y, axis=0) instead of A / B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmpi4hjhxyt.py:22
The behavior of expression A - B has been unified with elementwise_sub(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_sub(X, Y, axis=0) instead of A - B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmpi4hjhxyt.py:32
The behavior of expression A - B has been unified with elementwise_sub(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_sub(X, Y, axis=0) instead of A - B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmpi4hjhxyt.py:35
The behavior of expression A - B has been unified with elementwise_sub(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_sub(X, Y, axis=0) instead of A - B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\framework.py:2664: UserWarning: The Attr(force_cpu) of Op(fill_constant) will be deprecated in the future, please use 'device_guard' instead. 'device_guard' has higher priority when they are used at the same time.
  "used at the same time." % type)
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmpi4hjhxyt.py:68
The behavior of expression A * B has been unified with elementwise_mul(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_mul(X, Y, axis=0) instead of A * B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmp0klbcj5u.py:80
The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_add(X, Y, axis=0) instead of A + B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\tensor.py:658: UserWarning: paddle.assign doesn't support float64 input now due to current platform protobuf data limitation, we convert it to float32
  "paddle.assign doesn't support float64 input now due "
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmph7dypw3w.py:11
The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_add(X, Y, axis=0) instead of A + B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\layers\math_op_patch.py:341: UserWarning: C:\Users\DECOCL~1\AppData\Local\Temp\tmph7dypw3w.py:18
The behavior of expression A + B has been unified with elementwise_add(X, Y, axis=-1) from Paddle 2.0. If your code works well in the older versions but crashes in this version, try to use elementwise_add(X, Y, axis=0) instead of A + B. This transitional warning will be dropped in the future.
  op_type, op_type, EXPRESSION_MAP[method_name]))
2022-10-26 15:28:23,626 -  WARNING - No custom op iou3d_nms_cuda found, try JIT build
Compiling user custom op, it will cost a few seconds.....
'D:\Program' �����ڲ����ⲿ���Ҳ���ǿ����еij���
���������ļ���
2022-10-26 15:28:23,651 -    ERROR - iou3d_nms_cuda builded fail!
Traceback (most recent call last):
  File "E:/lyw/2022-8-12Detection/Paddle3D-develop/tools/export.py", line 69, in <module>
    main(args)
  File "E:/lyw/2022-8-12Detection/Paddle3D-develop/tools/export.py", line 64, in main
    model.export(args.save_dir, input_shape=args.input_shape)
  File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars.py", line 195, in export
    paddle.jit.save(self, save_path)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\jit.py", line 629, in wrapper
    func(layer, path, input_spec, **configs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\base.py", line 51, in __impl__
    return func(*args, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\jit.py", line 857, in save
    inner_input_spec, with_hook=with_hook)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 528, in concrete_program_specify_input_spec
    *desired_input_spec, with_hook=with_hook)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 436, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[cache_key]
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 801, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 790, in _build_once
    **cache_key.kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\base.py", line 51, in __impl__
    return func(*args, **kwargs)
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 740, in from_func_spec
    error_data.raise_new_exception()
  File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\error.py", line 336, in raise_new_exception
    six.exec_("raise new_exception from None")
  File "<string>", line 1, in <module>
RuntimeError: In transformed code:

    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars.py", line 116, in forward
	anchors_mask, batch_size)
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars_head.py", line 113, in post_process
	false_fn=lambda: self._box_empty())
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars_head.py", line 113, in post_process
	false_fn=lambda: self._box_empty())
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars_head.py", line 175, in _single_post_process
	false_fn=lambda: self._box_empty())
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars_head.py", line 175, in _single_post_process
	false_fn=lambda: self._box_empty())
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\detection\pointpillars\pointpillars_head.py", line 199, in _box_not_empty
	thresh=self.nms_iou_threshold)
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\models\layers\layer_libs.py", line 235, in rotate_nms_pcdet
        boxes = boxes.reshape([-1, 7])

        keep, num_out = iou3d_nms_cuda.nms_gpu(boxes, thresh)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        selected = order[keep[:num_out]]

    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 148, in __getattr__
	module = self._load_module()
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 134, in _load_module
	self.module = self.jit_build()
    File "E:\lyw\2022-8-12Detection\Paddle3D-develop\paddle3d\ops\__init__.py", line 122, in jit_build
	name=self.modulename, sources=sources, **args)
    File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\cpp_extension.py", line 857, in load
	_jit_compile(file_path, verbose)
    File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1112, in _jit_compile
	run_cmd(compile_cmd, verbose)
    File "D:\Program Files\Anaconda3\envs\paddle\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1154, in run_cmd
	compile, error))

    RuntimeError: Failed to run command: <built-in function compile>, errors: Command 'cd /d C:\Users\decocloudadmin\.cache\paddle_extensions && D:\Program Files\Anaconda3\envs\paddle\python.exe iou3d_nms_cuda_setup.py build' returned non-zero exit status 1.


Process finished with exit code 1

导出的命令是：
python tools/export.py --config configs/pointpillars/pointpillars_xyres16_kitti_car.yml --model outputs/pointpillars/iter_19800/model.pdparams --save_dir exported_model/PointPillars

@yuwen37 yuwen37 changed the title centerpoint训练报错 No custom op voxelize found, try JIT build centerpoint训练报错 ERROR - voxelize builded fail! PointPillars模型导出报错 ERROR - iou3d_nms_cuda builded fail! Oct 26, 2022
@FlyingQianMM
Copy link
Collaborator

麻烦提供下python、paddlepaddle_gpu、cuda、cudnn、显卡驱动、paddle3d的版本号。

@yuwen37
Copy link
Author

yuwen37 commented Oct 26, 2022

麻烦提供下python、paddlepaddle_gpu、cuda、cudnn、显卡驱动、paddle3d的版本号。

python==3.7
paddlepaddle-gpu ==2.3.1
cuda==10.2
cudnn==7.6.5
Driver Version: 441.66
paddle3d-develop

@FlyingQianMM
Copy link
Collaborator

你的windows上有装vs code吗？版本是多少呀？

@yuwen37
Copy link
Author

yuwen37 commented Oct 26, 2022

你的windows上有装vs code吗？版本是多少呀？

visual studio code ：
Version: 1.40.1 (system setup)
Commit: 8795a9889db74563ddd43eb0a897a2384129a619
Date: 2019-11-13T16:49:35.976Z
Electron: 6.1.2
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.18362

@FlyingQianMM
Copy link
Collaborator

FlyingQianMM commented Oct 26, 2022 •
edited
Loading

需要安装visual studio code 2019的ide

@yuwen37
Copy link
Author

yuwen37 commented Oct 26, 2022

需要安装visual studio code 2019的ide

是只能用这个ide跑吗，还是有别的什么作用？

@FlyingQianMM
Copy link
Collaborator

倒不是要在ide里面跑，只是需要cl.exe完成自定义算子的及时编译。需要借助visual studio，https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/custom_op/new_cpp_op_cn.html#jit-compile

@FlyingQianMM FlyingQianMM mentioned this issue Oct 27, 2022
Open
@ltj19900609
Copy link

需要安装visual studio code 2019的ide

微软的Visual studio 2019是IDE，而你说的Visual Studio code是微软推出的代码编辑器，并不是IDE，微软并没有Visual studio code 2019这个产品。

@Yutong-gannis
Copy link

请问解决了吗

@yuwen37
Copy link
Author

yuwen37 commented Feb 21, 2023

没有，后来没有再弄了

@nepeplwu nepeplwu assigned will-jl944 and unassigned FlyingQianMM Feb 1, 2024
@will-jl944
Copy link
Collaborator

需要安装visual studio code 2019的ide

微软的Visual studio 2019是IDE，而你说的Visual Studio code是微软推出的代码编辑器，并不是IDE，微软并没有Visual studio code 2019这个产品。

是Visual Studio 2019 （https://visualstudio.microsoft.com/zh-hans/vs/older-downloads/）

@yaobaishen yaobaishen mentioned this issue May 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@will-jl944 will-jl944

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@FlyingQianMM @ltj19900609 @yuwen37 @will-jl944 @Yutong-gannis