leaf node generation for real-world experiments

[gokul-gokz]

Hello
Great work.
I am curious on the following two aspects of the work in real-world use-case.

1. Validity of the tree and leaf nodes after placing an item:
   In real-world scenarios, the actual pose where the robot placed the object and the pose given by the policy might differ a little because of manipulation errors and noise. Additionally, there's also a chance that the current object being placed might move or disturb the previously placed objects. In these scenarios, some of the existing nodes in the tree might not be valid(as corner points might have moved by small amount) and so the whole tree structure need to be regenerated to represent the current state of the place bin accurately. How do you address this? As far as I understood from the paper(correct me if I am wrong), only the leaf nodes are removed if they are not valid but what about the internal & leaf nodes whose values are slightly off?

2. Orientation representation in state:
   While considering a new incoming item(Sx,Sy,Sz), if we want to consider the placement with more than one orientation for the item, how are these orientation values encoded into the state representation.

[alexfrom0815]

Hello, thanks for your attention!

1. If the position of the placed object is easy to shift, we can regenerate the leaf nodes from scratch each time instead of incrementally generating them. This is feasible and does not take too much time overhead.
2. The current version of the code supports orientation representation. It is integrated in the descriptor of the leaf node.

[gokul-gokz]

Let's take the scenario where the recently placed object has moved previously placed objects. Because of this the internal node set 'Bt' and the leaf node set 'Lt' is no longer valid anymore. If we want to regenerate the tree from scratch, we need to know exactly where did each and every object moved to create a correct spatial relation tree. To do this, we need to map each and every object in the camera image , to create the internal node set representation 'Bt' along with leaf node, which is not possible because we don't have information on each object to map them and create the tree, from the camera image.

So, how do you create a tree structure from scratch every-time maintaining the correct spatial relations?

[alexfrom0815]

Hello, this is indeed a very interesting question. However, I think it is not impossible to map each and every object in the camera image. In fact, based on our real robot experience, we can use the method of plane segmentation, or learning-based detection method, roughly determine the position of the observable box, and then correct the information of the internal node set 'Bt' and the leaf node set 'Lt', hope this can help.