Pipeline to print inferred 3D bounding boxes on images

[AlbertoRemus]

Hello! I find this work really interesting. After successfully testing inference (LMO and YCB) I was just interested in plotting the inference results as 3D bounding boxes on RGB images and by inspecting the code I bumped into:

GDR-Net/core/gdrn_modeling/gdrn_evaluator.py

Line 516 in 5fb30c3

def gdrn_inference_on_dataset(cfg, model, data_loader, evaluator, amp_test=False):

it is the function used for inference, which seems to show the results in terms of the different metrics, but not showing graphical results as I am looking for

In the same file I noticed the function:

GDR-Net/core/gdrn_modeling/gdrn_evaluator.py

Line 634 in 5fb30c3

def save_result_of_dataset(cfg, model, data_loader, output_dir, dataset_name):

which seems structurally similar but with some input differences, in particular I would like to ask if the input dataloader can be be computed for gdrn_inference_on_dataset as for save_result_of_dataset as in

GDR-Net/core/gdrn_modeling/engine.py

Lines 135 to 137 in 5fb30c3

	data_loader = build_gdrn_test_loader(cfg, dataset_name, train_objs=evaluator.train_objs)
	data_loader = self.setup_dataloaders(data_loader, replace_sampler=False, move_to_device=False)
	results_i = gdrn_inference_on_dataset(cfg, model, data_loader, evaluator, amp_test=cfg.TEST.AMP_TEST)

Since from preliminar debugging it seems it is not possible to access to the "image" field of the input sample in

GDR-Net/core/gdrn_modeling/gdrn_evaluator.py

Line 678 in 5fb30c3

images_ori = [_input["image"].clone() for _input in inputs]

Possibly related issue: #56

[wangg12]

If you want to access "image" during test, you need to change the dataloader

GDR-Net/core/gdrn_modeling/data_loader.py

Line 368 in 5fb30c3

if self.split != "train":

[Liuchongpei]

@wangg12 Your work is very meaningful. Can you tell me where the code for visualizing 6D pose is as shown in the figure below？

[AlbertoRemus]

If you want to access "image" during test, you need to change the dataloader

Many thanks, actually by inspecting the code, I realized that I don't need the image to be inside the image field (I could load it directly from disk), I think I would just need the 8-vertex bounding box in pixel coordinates to be returned among the results datastructure that at the moment contain only rotation and translation

[AlbertoRemus]

Many thanks, actually by inspecting the code, I realized that I don't need the image to be inside the image field (I could load it directly from disk), I think I would just need the 8-vertex bounding box in pixel coordinates to be returned among the results datastructure that at the moment contain only rotation and translation

With the modifications in my fork AlbertoRemus@7dd7e92 you can see what I was looking for (blue ground truth, red prediction):

Initially I hadn't realize that given:

• the 3D model
• matrices R, t (ground truth and predicted)
• matrix K intrinsic parameters

It is straightforward to compute the 2D projection of the 3D model points on the image plane, if one just needs the 8-vertex bounding box it just needs to form all the 8 possible combinations from {(x_min, x_max), (y_min, y_max), (z_min, z_max)}

[AlbertoRemus]

Provided the solution found in #64 (comment) I would close this issue (@wangg12 let me know if you need these modifications upstream I can open a PR)

@Liuchongpei your question is interesting, it may be worth to open a separate issue for that to visualize that plot with the segmentation mask

[AlbertoRemus]

#64 (comment)

@Liuchongpei your question is interesting, it may be worth to open a separate issue for that to visualize that plot with the segmentation mask

This quick and dirty modification should do the job: AlbertoRemus@38adf26

[cuge1995]

should

Hey, I can't find the branch you mentioned, does anything wrong?

[AlbertoRemus]

Hey, I can't find the branch you mentioned, does anything wrong?

Hi, now the links should be fixed!