[Bug] multiprocessing get stuck when creating waymo gt_database

[KickCellarDoor]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

1.1x branch https://github.com/open-mmlab/mmdetection3d/tree/1.1

Environment

I am using NGC pytorch container, pytorch 1.14

Reproduces the problem - code sample

python tools/create_data.py waymo --root-path ./data/waymo/ --out-dir ./data/waymo/ --workers 128 --extra-tag waymo

Most of the processes stuck when creating gt database. Creating gt database for Waymo takes about 24 days on a 56 cores machine.

Reproduces the problem - command or script

python tools/create_data.py waymo --root-path ./data/waymo/ --out-dir ./data/waymo/ --workers 128 --extra-tag waymo

Reproduces the problem - error message

No error message.

Additional information

Inside of the GTDatabaseCreater.create_single function,
point_indices = box_np_ops.points_in_rbbox(points, gt_boxes_3d) is used, which will use multi thread acceleration. So if you further use multiprocessing, most of the processes will stuck and the overall generating speed will be much slower than single thread.

Change

mmdetection3d/tools/dataset_converters/create_gt_database.py

Lines 613 to 617 in dfcf542

	multi_db_infos = mmengine.track_parallel_progress(
	self.create_single,
	((loop_dataset(i)
	for i in range(len(self.dataset))), len(self.dataset)),
	self.num_worker)

to

        multi_db_infos = mmengine.track_progress(
            self.create_single,
            ((loop_dataset(i)
              for i in range(len(self.dataset))), len(self.dataset)),
            )

will be much much faster. (24 days -> 5 hours)

[JingweiZhang12]

Thanks for your feedback, could you please create a PR to reduce the default number of workers and add docs about this dataset conversion?

[OrcunCanDeniz]

I was also getting stuck at numPointsInGtCalculater. As in #2364 adding below solved the issue so far.

import multiprocessing
multiprocessing.set_start_method('spawn')

[ammaryasirnaich]

I am also stuck in this part, did you manage to solve it?