I want to add action

[hyuDev]

hello, Thanks for awesome project(codes). It's really helpful to me.
I want to train navigation model with extra actions.
default action in your Ai2Thor is move, pickup, open_by_type,"rotate", "look", "stand", "crouch"...
I want to train new model, this model have a "push" or "knock",etc.
I think these actions can increase the accuracy, so I want to evaluate their performance by learning with my own algorithm. Where should these extra actions be added? I don't know what code to fix in your project.

[winthos]

Actions like Push and Pull already exist in our api. Please refer to them here.

Adding new actions directly to the codebase is a little more of an involved process. If something that is not currently supported by our api is needed, please feel free to describe it in as much detail as possible as a feature request that we can look into adding on our end.

If you want to add in custom actions yourself, I can respond with further details as needed.

[hyuDev]

oh,Thanks you.
I found Actions like Push and Pull already exist.
but I'm still having trouble learning using THoR. It seems to touch the wrong part of the code to learn the process of finding the desired object by avoiding objects without touching them. If I still want to add a new action, where should I add it?

[winthos]

It sounds like you are trying to do something where you want to find and avoid objects without having to push/pull or move them in some way in order to identify them? Before you start writing your own action, we do have some Environment Query actions available in our API that might help you.

There is the GetObjectInFrame action that allows you to query a position from the agent's screenspace, which will then return what objects are at that location without having to interact with them in a way that will require them to move.

Regardless of if the above action will help your purposes, I'll provide a quick breakdown of what you need to get started adding in your own actions if that is still something you want to do. First you need to find either the BaseFPSAgentController.cs script or the PhysicsRemoteFPSAgentController.cs script within the unity files. These should be located at unity/Assets/Scripts/... within the THOR directories.

You will want to add your custom action into the BaseFPSAgentController.cs file if you want the action to be available to all versions of the agent which can use either the default agent or a locobot version. Its likely that, if you are using the default agent mode, you will find it easier to just place a custom action in PhysicsRemoteFPSAgentController.cs as this inherits from the base class of BaseFPSAgentController.

For the sake of example, take a look at the implementation of GetObjectInFrame itself.

public void GetObjectInFrame(float x, float y, bool checkVisible = false) {
SimObjPhysics target = null;
screenToWorldTarget(
    x: x,
    y: y,
    target: ref target,
    checkVisible: checkVisible);
    if (target != null) {
        actionFinishedEmit(success: true, actionReturn: target.ObjectID);
    }

    if (target == null) {
        actionFinishedEmit(success: false, actionReturn: errorMessage);
    }
}

Note how the parameters for this action directly correspond with the input parameters if you were to call this action from the python controller

query = controller.step(
    action="GetObjectInFrame",
    x=0.64,
    y=0.40,
    checksVisible=False
)

This is the basic form for what you would need to do to create an action and call it from the python controller. The name of the action in python must match the function name, and any input parameters must also match.

When writing a custom action itself, it is important to end the action using actionFinishedEmit() when whatever you have written to happen in that action has finished. This lets the Unity side of the framework communicate that it is ready to emit a frame as well as metadata to the python controller. You can see in the example that you can pass in the success of whether the action failed or succeeded, which directly sets the metadata value of lastActionSuccess in event.metadata. Additionally, you can pass in any information you might want to return with actionReturn which also corresponds to actionReturn in event.metadata.. actionReturn can be used for pretty much anything, from in depth error messages to things like coordinates or other values.

To summarize, create a new action with the exact name of what you want the action to be called in either the BaseFPSAgentController.cs or PhysicsRemoteFPSAgentController.cs scripts. Make sure you end the action with an actionFinishedEmit() call once the action has either finished or failed, otherwise the Unity client will hang.

Depending on exactly what kind of action you are trying to make, there are some other more minute details that may require further explanation. If you are doing something that takes a sim object as a target, there are a few ways to go about it. Similarly, if an action involves any aspect of agent movement or manipulation like moving the agent hand or moving the agent's position, that also has some extra thing to consider. Please post more details to this discussion if you end up needed more of those details, as they are too intricate to quickly detail for now.