Add multiprocessing for eval_bop19_pose.py #110

nv-nguyen · 2023-12-01T20:35:05Z

Adding multiprocessing with 10 workers by default for evaluation script eval_bop19_pose.py. The original script can still be run with argument --num_worker 1.

I made some unit tests with MegaPose's results.

Here are the run-time comparisons with and without multiprocessing. Note that the scores remains unchanged. The run-time improvement is more significant for large datasets such as T-LESS.

I can do some refactoring to make the code cleaner after this pull-request is accepted.

@thodan @MartinSmeyer @ylabbe

…ose.py

…uation script

nv-nguyen · 2024-01-15T15:02:39Z

To test the multiprocessing:

bash bop_toolkit_lib/tests/unit_test.sh

thodan

Please have a look at my comments.

thodan · 2024-01-12T20:12:43Z

scripts/eval_calc_errors.py

-        scene_errs = []
-
+        # collect all the images and their targets
+        im_metaDatas = []


Please follow the snake case style, e.g., let's rename "im_metaDatas" to "im_meta_datas" or just "im_metadatas".

thodan · 2024-01-17T10:13:56Z

bop_toolkit_lib/tests/unit_test.sh

@@ -0,0 +1,21 @@
+#!/bin/bash


Please change the name of this file to reflect what it is testing (e.g. "eval_bop19_pose_test.sh").

thodan · 2024-01-17T10:16:13Z

scripts/enumerate_test_targets.py

@@ -0,0 +1,107 @@
+# Author: Tomas Hodan (hodantom@cmp.felk.cvut.cz)
+# Center for Machine Perception, Czech Technical University in Prague
+


Please add a docstring at the beginning of this script to briefly describe what this script does ('''docstring''', such as in https://github.com/thodan/bop_toolkit/blob/master/scripts/calc_gt_info.py).

thodan · 2024-01-17T10:18:55Z

bop_toolkit_lib/tests/unit_test.sh

+
+# Loop through each file name and execute the command
+for FILE_NAME in "${FILE_NAMES[@]}"; do
+    python scripts/eval_bop19_pose.py --renderer_type=vispy --results_path $INPUT_DIR --eval_path $INPUT_DIR --result_filenames=$FILE_NAME --num_worker 10 >> $OUTPUT_FILE 2>&1


A unit test should ideally compare the output with an expected output. Would it be possible to add such a comparison (i.e. compare the content of the output files in this case)? Now it is not clear how to learn that the test run was successful (other than it didn't crash).

thodan · 2024-01-17T10:21:52Z

bop_toolkit_lib/tests/README.md

@@ -0,0 +1,9 @@
+Simple tests for using multiprocessing with the BOP toolkit.


I would change this line to "# Unit tests of the BOP toolkit" as the test below doesn't test just multiprocessing and we may want to add more tests in the future. I would then add a subsection "## Test of eval_bop19_pose.py" above the below command and table.

thodan · 2024-01-17T10:26:53Z

bop_toolkit_lib/renderer_batch.py

+        for worker_id in range(self.num_workers):
+            cmd = [
+                "python",
+                "bop_toolkit_lib/call_renderer.py",


Is parallel calling of this call_renderer script (which always needs to initiate the rendering buffer, load object model, etc. from scratch) really faster than just running a single renderer (that can be initialized only once at the beginning; as before this diff) in sequence? If not or if the difference is not significant (which I assume) I would stick to calculating VSD in sequence (reason: the code is simpler to understand and maintain).

As shown in the table above, the run-time difference is significant for BOP5 datasets. This is because (1) the initialisation is not very long (<10s) while all the workers are run in parallel, and (2) the number of images to render is much larger than the number of objects to initialize + the renderer buffer . However, I got your points so I add this check which will make sure that the multiprocessing is done when it is need.

To make the code more readable and modular, it would be great if we could clearly divide the renderer and the error computation code. For example,

defining POSE_ERROR_VSD_ARGS in bop_toolkit_lib/pose_error.py

rename call_renderer.py to call_vsd_worker.py

pass the batch_renderer to pose_error.vsd(..) instead of other renderers ren, and then compute vsd error in that function based on the renderer argument type.

In my latest commit, I updated with your reviews except for the last point regarding batch renderer. As we discussed last week, I initially attempted to do exactly what you suggested but unfortunately, none of the renderers in the toolkit allow multiprocessing. Therefore, I had to find an alternative solution by running call_vsd_worker.py n_workers times.

nv-nguyen · 2024-01-17T13:18:47Z

I resolved all reviews and add a comparison with expected outputs. Here is the output in my terminal when running the unit tests (I removed unit tests of YCB-V and T-LESS to make it fast since these two datasets are quite large):

…itives for benchmarking

nv-nguyen · 2024-02-19T06:37:12Z

I fixed a bug and finalised here the scoring function for 6D detection task. The main difference with the scoring function in 6D localization task is this part where we count the number of false positives to calculate precision.
The final mAP score for 6D detection task is calculated across all object IDs here (similar to 2D detection task).

To test eval_bop24_pose.py script:
python bop_toolkit_lib/tests/eval_bop24_pose_test.py

To test eval_bop24_pose.py script with "fake" 10.000 false positives:
python bop_toolkit_lib/tests/eval_bop24_pose_test.py --num_false_positives 10000

MartinSmeyer

@nv-nguyen I'm not yet done with the review, but feel free to already address the first comments.

bop_toolkit_lib/score.py

bop_toolkit_lib/renderer_batch.py

MartinSmeyer · 2024-04-05T16:49:07Z

bop_toolkit_lib/renderer_batch.py

+        for worker_id in range(self.num_workers):
+            cmd = [
+                "python",
+                "bop_toolkit_lib/call_renderer.py",


To make the code more readable and modular, it would be great if we could clearly divide the renderer and the error computation code. For example,

defining POSE_ERROR_VSD_ARGS in bop_toolkit_lib/pose_error.py

rename call_renderer.py to call_vsd_worker.py

pass the batch_renderer to pose_error.vsd(..) instead of other renderers ren, and then compute vsd error in that function based on the renderer argument type.

bop_toolkit_lib/renderer_batch.py

MartinSmeyer · 2024-04-05T16:51:58Z

scripts/eval_calc_errors.py

+                                if p["num_workers"] == 1:
+                                    e = pose_error.vsd(
+                                        R_e,
+                                        t_e,
+                                        R_g,
+                                        t_g,
+                                        depth_im,
+                                        K,
+                                        p["vsd_deltas"][dataset],
+                                        p["vsd_taus"],
+                                        p["vsd_normalized_by_diameter"],
+                                        models_info[obj_id]["diameter"],
+                                        ren,
+                                        obj_id,
+                                        "step",
+                                    )
+                                else:  # delayed the calculation of the error for renderer_batch
+                                    e = renderer_batch.POSE_ERROR_VSD_ARGS(
+                                        R_e=R_e,
+                                        t_e=t_e,
+                                        R_g=R_g,
+                                        t_g=t_g,
+                                        depth_im=depth_im,
+                                        K=K,
+                                        vsd_deltas=p["vsd_deltas"][dataset],
+                                        vsd_taus=p["vsd_taus"],
+                                        vsd_normalized_by_diameter=p[
+                                            "vsd_normalized_by_diameter"
+                                        ],
+                                        diameter=models_info[obj_id]["diameter"],
+                                        obj_id=obj_id,
+                                        step="step",
+                                    )



So here it would be nice to just call pose_error.vsd(..) with either ren or renderer_batch and then call the vsd error computation inside.

MartinSmeyer · 2024-04-11T14:50:36Z

bop_toolkit_lib/tests/eval_bop24_pose_test.py

+# Define the dataset dictionary
+FILE_DICTIONARY = {
+    # "icbin": "cnos-fastsammegapose_icbin-test_7c9f443f-b900-41bb-af01-09b8eddfc2c4.csv",
+    # "tudl": "cnos-fastsammegapose_tudl-test_1328490c-bf88-46ce-a12c-a5e5a7712220.csv",
+    "lmo": "cnos-fastsammegapose_lmo-test_16ab01bd-f020-4194-9750-d42fc7f875d2.csv",
+    # "ycbv": "cnos-fastsammegapose_ycbv-test_8fe0af14-16e3-431a-83e7-df00e93828a6.csv",
+    # "tless": "cnos-fastsammegapose_tless-test_94e046a0-42af-495f-8a35-11ce8ee6f217.csv",
+    # Add more entries as needed
+}


Could we maybe run a last sanity test and check whether we get 0mAP with empty predictions and 100mAP with ground truth predictions?

Yes, I agree this is important. I added in my latest commit the sanity test with ground-truth predictions.

For empty predictions, maybe you mean “totally wrong” predictions instead of “empty” since the current code will already raise some errors when the predictions are empty anyway.

MartinSmeyer · 2024-04-11T14:53:36Z

scripts/eval_calc_errors.py

+                                        "step",
+                                    )
+                                else:  # delayed the calculation of the error for renderer_batch
+                                    e = renderer_batch.POSE_ERROR_VSD_ARGS(


Suggested change

e = renderer_batch.POSE_ERROR_VSD_ARGS(

e = pose_error.POSE_ERROR_VSD_ARGS(

Updated. Thanks!

MartinSmeyer

lgtm

nv-nguyen added 6 commits November 29, 2023 17:30

Adding multiprocessing (10 workers) for scoring parts in eval_bop19_p…

b261636

…ose.py

Merge branch 'master' of https://github.com/nv-nguyen/bop_toolkit

cb19aa3

Add multiprocessing for calcul_errors(except for VSD)

e9d4909

Fix bug in calculates_error, add batch_renderer, logger

3e9d84f

Change logger to log files instead of time and disable output of eval…

ec6e4ac

…uation script

Merge branch 'thodan:master' into master

2fa02e7

nv-nguyen added 2 commits January 15, 2024 17:23

Add unit tests

873248a

Add readme for unit tests

d103654

thodan reviewed Jan 17, 2024

View reviewed changes

nv-nguyen added 5 commits January 17, 2024 15:41

Resolve Tomas's reviews (add score verifications)

0d21d4f

add script to subsample test set

a3370f7

reformat scripts/subsample_dataset.py

69a56ba

Add script eval_bop24_pose.py to evaluate 6D object detection task

a4cbc2f

Fix bug in counting false positives and scripts to generate false pos…

888ec88

…itives for benchmarking

MartinSmeyer reviewed Apr 5, 2024

View reviewed changes

refactoring code with Martin's reviews

2e69652

MartinSmeyer reviewed Apr 11, 2024

View reviewed changes

Add unit test with predictions generated from gt

7e063f2

MartinSmeyer approved these changes Apr 26, 2024

View reviewed changes

Update requirements.txt

98a9de8

nv-nguyen merged commit 2fe516b into thodan:master Apr 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multiprocessing for eval_bop19_pose.py #110

Add multiprocessing for eval_bop19_pose.py #110

nv-nguyen commented Dec 1, 2023 •

edited

Loading

nv-nguyen commented Jan 15, 2024

thodan left a comment

thodan Jan 12, 2024

thodan Jan 17, 2024

thodan Jan 17, 2024

thodan Jan 17, 2024

thodan Jan 17, 2024

thodan Jan 17, 2024

nv-nguyen Jan 17, 2024

MartinSmeyer Apr 5, 2024

nv-nguyen Apr 18, 2024

nv-nguyen commented Jan 17, 2024

nv-nguyen commented Feb 19, 2024

MartinSmeyer left a comment

MartinSmeyer Apr 5, 2024

MartinSmeyer Apr 5, 2024

MartinSmeyer Apr 11, 2024

nv-nguyen Apr 18, 2024

MartinSmeyer Apr 11, 2024

nv-nguyen Apr 18, 2024

MartinSmeyer left a comment

		@@ -0,0 +1,107 @@
		# Author: Tomas Hodan (hodantom@cmp.felk.cvut.cz)
		# Center for Machine Perception, Czech Technical University in Prague

		@@ -0,0 +1,9 @@
		Simple tests for using multiprocessing with the BOP toolkit.

	e = renderer_batch.POSE_ERROR_VSD_ARGS(
	e = pose_error.POSE_ERROR_VSD_ARGS(

Add multiprocessing for eval_bop19_pose.py #110

Add multiprocessing for eval_bop19_pose.py #110

Conversation

nv-nguyen commented Dec 1, 2023 • edited Loading

nv-nguyen commented Jan 15, 2024

thodan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nv-nguyen commented Jan 17, 2024

nv-nguyen commented Feb 19, 2024

MartinSmeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MartinSmeyer left a comment

Choose a reason for hiding this comment

nv-nguyen commented Dec 1, 2023 •

edited

Loading