Not detecting microwave during data generation

[RishiHazra]

Hi,

I'm trying to generate data with new goal definitions. I've updated the code to work on the AI2Thor 4.2.0. While it's able to generate new data for goal pertaining to pick, place, clean and cool -- when it comes to heat sub-goal, it no longer works. While trying to narrow down the issue, I realized that self.env.last_event.metadata["objects"] (in gen/game_states/task_game_state_full_knowledge.py) does not show "Microwave" even though it is present in the scene and in clear view. It can detect "StoveBurner" & "Fridge" though.

As such the planner keeps failing with the error:

ff: goal can simplified to FALSE. No plan will solve it

Any help is appreciated. Thanks !!!

[askforalfred]

Hey Rishi, that's exciting! I hope you're able to find a workaround, but we feel that this particular issue is outside the scope of the maintenance of the ALFRED codebase. My gut feeling is "Microwave" might either be named something else in the newer version in some (all?) scenes [fixable on your end] or the mesh isn't named in the metadata in a particular scene [open issue on AI2-THOR repo if you can confirm]. Good luck!

…

On Thu, Aug 11, 2022 at 6:07 PM 'RishiHazra' via askforalfred < ***@***.***> wrote: Hi, I'm trying to generate data with new goal definitions. I've updated the code to work on the AI2Thor 4.2.0. While it's able to generate new data for goal pertaining to pick, place, clean and cool -- when it comes to heat sub-goal, it no longer works. While trying to narrow down the issue, I realized that self.env.last_event.metadata["objects"] (in gen/game_states/task_game_state_full_knowledge.py) does not show "Microwave" even though it is present in the scene and in clear view. It can detect "StoveBurner" & "Fridge" though. As such the planner keeps failing with the error: ff: goal can simplified to FALSE. No plan will solve it Any help is appreciated. Thanks !!! — Reply to this email directly, view it on GitHub <#121>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN3HWRF46AZW43ORB7IK2IDVYWPWPANCNFSM56KBTNTA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> -- You received this message because you are subscribed to the Google Groups "askforalfred" group. To unsubscribe from this group and stop receiving emails from it, send an email to ***@***.*** To post to this group, send email to ***@***.*** To view this discussion on the web visit https://groups.google.com/d/msgid/askforalfred/askforalfred/alfred/issues/121%40github.com <https://groups.google.com/d/msgid/askforalfred/askforalfred/alfred/issues/121%40github.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout.

[RishiHazra]

Thanks for the quick response. I can confirm that Microwave hasn't been renamed in the new version of AI2Thor.

On digging further, I was able to trace it to the function in alfred/env/thor_env.py

def restore_scene(self, object_poses, object_toggles, dirty_and_empty):
        '''
        restore object locations and states
        '''
        super().step(dict(
            action='Initialize',
            gridSize=constants.AGENT_STEP_SIZE / constants.RECORD_SMOOTHING_FACTOR,
            renderImage=constants.RENDER_IMAGE,
            renderDepthImage=constants.RENDER_DEPTH_IMAGE,
            renderClassImage=constants.RENDER_CLASS_IMAGE,
            renderObjectImage=constants.RENDER_OBJECT_IMAGE,
            visibility_distance=constants.VISIBILITY_DISTANCE,
            makeAgentsVisible=False,
        ))

        

        if len(object_toggles) > 0:
            # TODO: problem here: the API has change on these two attributes.
            for o in object_toggles:
                super().step((dict(action='SetObjectStates', 
                                SetObjectStates=o)))
        
        if dirty_and_empty:
            # TODO: problem here: the API also change on these two attributes.
            for o in object_poses:
                super().step(dict(action='SetObjectStates',
                    SetObjectStates={'objectType': o['objectName'].split('_')[0], 'stateChange': 'dirtyable', 'isDirty': True}))

                super().step(dict(action='SetObjectStates',
                    SetObjectStates={'objectType': o['objectName'].split('_')[0], 'stateChange': 'canFillWithLiquid', 'isFilledWithLiquid': False}))
        
        super().step((dict(action='SetObjectPoses', objectPoses=object_poses)))

It seems like running this function with action='SetObjectPoses' leads to the discarding of certain objects -- as in Microwave is detected initially in the list of objects in the event metadata (last_event.metadata["objects"]), however on running this function, it gets discarded. Moreover, it also discards objects like GarbageCan, Shelf, etc all of which are initially detected in the event metadata. This happens irrespective of the FloorPlan. I can see a couple of TODO marked there which sort of resembles my problem.

I've opened an issue on AI2Thor repo too.

[RishiHazra]

Update: I was able to figure out the error. By editing the object_poses, to include 'moveable' objects as well as 'pickupable' objects, I was able to resolve the issue. Apparently, objects like Microwave, CoffeeMachine, GarbageCan, etc. have moveable=True property and while performing the action='SetObjectPoses', AI2Thor drops all objects that aren't moveable or pickupable (see documentation)

object_poses = [{'objectName': obj['name'].split('(Clone)')[0],
                                        'position': obj['position'],
                                        'rotation': obj['rotation']}
                                        for obj in env.last_event.metadata['objects'] if obj['pickupable'] or obj['moveable']]

Going ahead and closing the issue.