New form of user input in the visualizer: context menus

[thaije]

Is your feature request related to a problem? Please describe.
Part of being a successful team depends on being able to effectively and efficiently communicate. As of now, communication between team members can only be done with the free-text chat. However, this makes it difficult for the human agent to task another agent in doing a specific action on a specific object: it is slow (have to write it all out), prone to error (have to type the agent ID, action name and object ID correct), and results in complicated agents (with natural language processing capabilities) or very brittle agents that fail with a single typo.

A second issue is that some of these difficulties are also encountered for a user when controlling their human agent. At the moment, actions are tied to specific keys. Issues with this are:

• It scales badly. In a complicated scenario with a large variety of objects and actions, each action has to have a specific key.
• It can become difficult for the test subject to remember all the controls. This is exacerbated by the often short training period before experiments in which test subject can familiarize themselves with the controls. Even more so as experiments often introduce stress-inducing factors, which do not have as goal to study the test subject forgetting how to control its avatar.

Describe the solution you would like
A context menu that pops up when right clicking on an object that can be filled with possible actions that the human agent can execute.

This menu can be used in various ways:

• Tasking the human agent. Right click on an object shows the possible actions for the human agent.
• Tasking another agent. Select another agent (left click), and then open the context menu (right click) on another object. Now the actions should be shown that the selected agent can be tasked with performing on the selected object. For instance, picking up a block. Or moving to a certain location (background tiles can also be selected).

Describe alternatives you have considered
Using the keyboard which is currently implemented. However, as described above, this has some disadvantages.

Additional context
The front-end already has code implemented for opening a context menu. However, any filling of that menu or functionality on the side of MATRX has not yet been implemented.

Possible future extensions of this feature:

• Select multiple agents to command them at once.
• Implement some sort of permission system, that specifies who can task who, or with which specific actions. (E.g. Can only task agents from your own team).

[thaije]

A way in which this feature could be implemented:

A new function is added to the AgentBrain class:
function fill_tasking_menu(requester_agent_ID, target_obj_ID).

The requester_agent_ID is the ID of the agent that requested the fill_tasking_menu. The target_obj_ID is the ID on which the context menu was opened. The latter can also be a location (in the frontend a background tile).

The function returns a list with menu items, with each menu item containing:

• action_display_name: the name that should be displayed in the menu.
• action_class_name: the class name that should be sent with the API request when the user clicks on the menu item.

Where this action<->object mapping comes from might be similarly defined to the key_action_map already implemented for key<->action pairs, but now for object(type?)<->action pairs. It can also be overwritten by users to do some fancy computation on what actions should be permitted.

Furthermore, the fill_tasking_menu should have a corresponding function in the agent_avatar object, such that it can also be accessed by the gridworld.

The overal flow then could go something like this:

• In the frontend, the user right clicks on an object.
• The click event is caught, the obj ID is fetched from the object on which was clicked, and an async API request is sent to the API server at address api_url:3001/fill_context_menu with the object ID and current agent ID as parameters.
• The API receives the request, and asynchronously calls: gridworld.registered_agents[requester_agent_ID].fill_tasking_menu(requester_agent_ID, target_obj_id), which returns the above specified output that is passed back to the frontend as a reply to the API request.
• The frontend fills the menu with the received data, and attaches on click listeners to the options that sent a new request to the API on click, passing the user input.

The case for tasking other agents is almost identical, except for that there are 3 variables passed from the visualizer to the API:

• The requester_agent_ID: the ID of the human agent being controlled
• The target_agent_ID: the ID of the agent that was selected
• The target_obj_ID: the ID of the object on which the context menu was opened.

The API then calls the fill_tasking_menu of the target_agent_ID.

This approach has as advantages:

• It is asynchronous, so it is fast, even at slow tick_durations.
• Clear and uniform approach that works for both cases described above.
• In line with our coding approach so far (?).

Disadvantages:

• The API has to have access to the grid_world object, such that it can access the fill_tasking_menu function from the API without waiting for ticks to pass.
• The OODA or sense-decide-act loop might become more diffuse with asynch functionality, especially if the fill_tasking_menu can be overwritten by users and they do some fancy computation in there.

@jwaa , what do you think of the approach above? This week is the Co-active teaming sprint, so I want to have it done before the end of the week :)

[jwaa]

@thaije; I like the fact that the user's side is in the AgentBrain. A place where the user already has stuff to do anyway.

Though I also think this is only part of the solution. Things I still miss:

1. How does one transform a 'context-menu-click-action' such as "move to ..." into a series of actions? Instead of outputting a single Move action, this would require the usage of our path finding functionality. Other custom things may even cause more trouble, such as "Guard this area". I can imagine that these kind of things do not directly output an action, but instead set a certain variable in the AgentBrain, which the decide_on_action takes into account.

2. I do not quite understand yet where the action logic happens. You mention a mapping from object <-> action. Is this object the name of the context menu item that was clicked? Is it something else? Also, the fill_tasking_menu seems to return a map of name<->action presuming that it only fills the context menu but not actually do the logic. The suggestion seems to imply it also performs the logic, but where does this return to then? And what is returned actually?

3. If I understand correctly (and again, this arrives from me not fully understanding where the action logic/selection happens), a context menu action happens parallel from the world ticks? Since the action is decided and executed not in the decide_on_action method? This might cause some conceptual turn-taking problems.

4. A minor one; the neccessity of requester_agent_ID is a bit unclear to me. If I understand correctly, the API makes sure that the correct agent's fill_tasking_menu is always called. So in this case (inside the AgentBrain), the self.agent_id would also be that ID making that parameter obsolete.

A suggestion:
Keep the fill_tasking_menu as mentioned, but change the mechanism of how the action is performed. Let this function simply return a dictionary of the name of the context menu item and a linked function object of a predefined signature.

When a certain context menu item is clicked, the frontend returns the menu item's name and the context (who clicked, what was selected, what was clicked upon) and the API finds the appropriate function object based on the agent who clicked and the menu's item name.

Next, that function is performed and given the location on which was clicked, the id of what was selected and what was clicked upon (location is given because we cannot garuantee that there is always an ID available, e.g. there might exist none there, or it is not within a agent's state). These functions should return None. This to force people making these functions not to actually return actions or behaviour, but to change the internals of the AgentBrain to do something next tick (e.g. move, pick up, send message, alter property, etc.).

An example of this for a context menu containing a "Move to..." and "Pick up...".

class MyHumanBrain(HumanAgentBrain):
    def __init__():
        self.pick_up_obj = None
        self.move_to_loc = None

    ....

    def fill_tasking_menu(self, selected_obj_id, target_obj_id)
        requester_agent_ID =  self.agent_id
        context_menu = {
            "Move to...": self.context_move_to
            "Pick up...": self.context_pick_up
       }

    def context_move_to(self, clicked_location, target_agent_id, target_obj_id):   # MATRX user method
        if target_agent_id is None or target_agent_id == self.agent_id):
            self.move_to_loc = clicked_location
        else:
            mssg = Message({"move_to": clicked_location})
            self.send_message(to=target_agent_id, message=mssg)

    def context_pick_up(self, clicked_location, target_agent_id, target_object_id):   # MATRX user method
        if target_agent_id is None or target_agent_id == self.agent_id):
            self.move_to_loc = location
            self.pick_up_obj = target_object_id
        else:
            mssg = Message({"pick_up": clicked_location, "location": clicked_location})
            self.send_message(to=target_agent_id, message=mssg)

    def decide_on_action(state):
        ...

        self.parse_context_messages()

        action, kwargs = self.handle_context_move_to()
        if action is not None:
            return action, kwargs

        action, kwargs = self.handle_context_pick_up()
        if action is not None:
            return action, kwargs


    def parse_context_messages():   # MATRX user method
        # Parse received message on a 'pick up' and/or 'move_to' messages, and set the brain's variables

    def handle_context_move_to():   # MATRX user method
        if move_to_loc is not None:
            # Apply the pathfinder to move to that location, set `move_to_loc` to None when done

    def handle_context_pick_up():   # MATRX user method
        if pick_up_obj is not None:
            # Return the `grab_object` action with the `pick_up_obj` id.

The API then only has to parse the incoming clicks and check the menu item's name in the human agent's brain and perform the function object. The actual logic part is dealt with in the decide_on_action method of this agent. Furthermore, it utilizes the message structure we have in MATRX to communicate actions to selected agents (them being selected and then opening the context menu on another agent or object).

A major disadvantage of this approach is that the MATRX user has to know about function objects and have the awareness how these are callbacked. Which might require a very clear tutorial with examples. Even then it might make debugging a bit more difficult, but I find this hard to estimate.

However, it still has all the advantages you mentioned @thaije. In addition it relies even more on the way people work with MATRX (decide on stuff in one method) and its functions (messaging). In addition it has the added advantage that we can easily extend the existing HumanAgentBrain with many potential menu items a MATRX user can simply turn on or off through some constructor variable. Also, MATRX users can use such predefined methods to populate their own context menus. For example all the methods tagged with # MATRX user method can be already given to a MATRX user.

[thaije]

Hey @jwaa, looks like a good starting point! I fully agree with most of your points.
I would like to suggest to rename the function from fill_tasking_menu to fill_context_menu, as the context menu can be used for more than only tasking.

Also, the only which wasn't exactly clear yet was the connection between the human agent brain and how the context menu info propagates to the visualizer via the API and back. Starting with how the API calls the fill_context_menu, do you agree that we have to have access to the grid_world in the API, and access to the fill_context_menu function via the agent avatar? I would suggest we implement it in similar fashion to existing functions such as the set_messages_func.

Secondly, how did you have in mind for the API<->frontend connection to work with passing the context menu functions? From what I understand this is what the API passes to the frontend to fill the context menu with:

context_menu = {
            "Move to...": self.context_move_to,
            "Pick up...": self.context_pick_up
       }

Then what should the frontend send back to the API when a menu item is clicked? The function pointer? I don't think that will work, as the API converts any sent data to json/text, such that the self.context_move_to function pointer becomes a string. We would thus first have to convert a string then to the function context_move_to of a specific agent (which self) was pointing to, however I don't know yet how we can achieve that.

[thaije]

• No callback spaghetti
• Intuitive & easy to use for the user

[thaije]

Okay so here a new shot at this to-be-implemented feature.

Requirements

What do we want this feature to do

See the example grid below: in this example there is a human agent, an artificial agent, and two different types of objects.
The context menu will provide a menu with options for the human agent, that can be opened by right clicking on any of the objects. See image 2.

Clicking on these options might trigger certain actions, sending of messages, or trigger other custom code. The context menu can be either used to send input to the human agent that is being controlled (I want to do X), or task other agents by first selecting that agent and than right click on an object (go do X to object Y).

Thus, there are two methods of using this new feature:

• To communicate something to the human agent being controlled (e.g. do a specific action, send a message, or something else)
• To request something from another agent, on behalf of the human agent being controlled (e.g. can you do action X, send a message, or do something else).

What we want: Types of options for the context menu

We should be able to use the context menu for a variety of things. A number of options that the context menu should be able to display:

• Do action X on this on object (or request that another agent does this)
• Send a message (to a specific agent on which the menu was opened)
• Do a meta-action: an action that consists of multiple sub actions. E.g. move to this location, a command that consists of a sequence of Move actions.
• Other custom things. E.g. execute a specific function in the human agent when that option is clicked.

Conditionality of specific context menu options

Imagine method 1 of using this feature: we rightclick on an object to open a menu, and clicking on options in that menu should make the human agent do things. In this case, which specific options should be listed? These specific options can depend on these factors:

• Properties or type of the object on which was right clicked. E.g. A ball object might have the PickUp action as a option, a wall doesn't.
• Properties of the gridworld. E.g. a certain event occurs at tick 10, which prevents anyone from doing certain things.
• Properties of the human agent. E.g. a human agent that doesn't have wheels or legs cannot do the move action, thus that is not an option.

Now image method 2 of using this feature: we first select another agent, and then right click on an object. We want to use the context menu to request the selected robot to do certain things on
behalf of the human agent. Which specific options should be listed?
These specific options can depend on these factors:

• All the things listed for method 1, but from the robots standpoint:
  • Properties of the type of the object on which was right clicked
  • Properties of the gridworld
  • Properties of the robot.
• Permissions, what is the human agent allowed to task/say/communicate to the robot. E.g. the robot might have an action that only certain people can command the robot to do.

Implementation

The most important thing is for this feature to be ease-to-use, while still facilitating as much of the functionality stated above as possible. The following will change in MATRX, or is to be added.

The frontend <-> backend communication flow

Every agent and human agent should have a new function:

fill_context_menu(self, requester_agent_ID, selected_object_ID, selected_location)

When in the front-end the user opens a context menu by clicking on an object, an async request is sent to the API with
some parameters (current_agent_ID, selected_agent_ID, selected_object_ID, selected_location). In the case the user does not select another agent, and opens the context menu on an empty tile these parameters are sent:

current_agent_id = ID of the human agent currently being controlled
selected_agent_id = None 
selected_object_id = None 
selected_location = location of the tile on which the context menu was opened

The API receives the request with these parameters, and calls the fill_context_menu function from the desired agent. The desired agent being: if any agents where selected, that agent, otherwise the human agent that was being controlled.

The fill_context_menu function returns an json array with options, see below in the next section on how that is done.
The API sends back this json list, which is used by the frontend to display the options. The context options json list roughly looks like this:

{[ 
    {"OptionText": Option1,
     "OptionType": ..., 
     "OptionSpecificData": ....
    },
    {"OptionText": Option2,
     "OptionType": ..., 
     "OptionSpecificData": ....
    },
    etc.
]}

When the user now clicks on one of the options, an new userinput API request is sent, with as data the optionType and optionSpecificData.
This data is received by the human agent as with any other userinput: in the decide_on_action function in the userinput variable.

From there, the user can decide on what to do with it.
To make it easier for the user, a number of default context menu options (Actions and Messages) are processed automatically, as shown in the next section.

Generating options for a context menu

There are three types of options that can be used to fill the context menu:

• Action: When the user clicks on this. The agent should perform a MATRX action.
  The json for such as an action looks like this:

  "OptionText": action.__name__,
  "OptionType": "action",
  "ActionName": action.__name
  "ActionKwargs": {obj_id: selected_object_ID, arg2_for_action: 2, ..}
  

  (Note: you can fill in anything you like as the OptionText)

• Message: Clicking on such an option makes the current human agent send a specific message. E.g. a global message that an object has been found, or sending a message to another agent. The json looks roughly like this:

  "OptionText": "Hey other agent! Can you help me?",
  "OptionType": "message",
  "MessageParameters": {"from": current_agent_ID, "to": selected_object_ID, "content": "Hey other agent! Can you help me?"}
  

• Custom: for all other things that don't fit in the above categories.
  E.g. a meta-action or sending custom messages.
  An example json for a context menu option that can be used to initiate a meta-action:

  "OptionText": "Bring this victim to the command post",
  "OptionType": "custom",
  "CustomParameters": {"function_name": "bring_victim_to_cp"}
  

  Another example json for sending a message of a custom Message type:

  "OptionText": "Hey other agent! Can you help me?",
  "OptionType": "custom",
  "CustomParameters": {"function_name": "send_custom_message", "from": current_agent_ID, "to": selected_object_ID, "content": "Hey other agent! Can you help me?", "custom_var1": ...}
  

Parsing context menu userinput

If a context menu option of type Message is clicked, an API request is done to the already existing /send_message API handler, that sends the received message.
If a context menu option of type Action is clicked, this is filtered by a few lines of simple script that can be added to the default human agent:

function decide_on_action(self, state, ..., userinput):
    # process the first userinput 
    if userinput is not None:
        userinput = userinput[0]

    # check if we received userinput in the form of an action we need to do, and do it if so
    if "OptionType" in userinput and userinput["OptionType"] == "action": 
        return userinput["action"], userinput["ActionKwargs"] 

    ...

If a context menu option type Custom is clicked, this can be caught by the user in the decide_on_action as well. But they can decide how to further process this themselves.

Summary

This solution provides a context menu that is easy to use for beginners, but with enough options for more advanced uses. Context menu's can be filled based on a large number of properties, but are already useful and working with a few lines of code.
For example, if a user wants a context menu with all actions the agent can do, it's as simple as:

def fill_context_menu(self, requester_agent_ID, selected_object_ID, selected_location):
    context_menu = [] 
    
    for action in self.possible_actions: 
        context_menu.append({
            "OptionText": action.__name__,
            "OptionType": "action",
            "ActionName": action.__name
            "ActionKwargs": {"obj_ID": selected_object_ID}
        })
    
    return json.encode(context_menu)

The code above can be easily extended to add checks on which actions specifically are possible for an object, or adding the code above to an agent such that it can be tasked by the human agent.

[jwaa]

Thanks for the elaborate description! Really helpful!
I do have some questions though, most regard the scope of this functionality as it sounds quite ambitious and perhaps we should start a bit simpler.

1. General remark (and the most important one I have): To improve the ease-of-use, why not try and utilize the message structure we already have in MATRX? In your description you already mention the idea of sending a message with a context menu. Why not do it like this all the time? If a human opens the context menu with nothing selected and clicks the "Move to..." option, a message (defined in the dict that fill_context_menu returns) is send to the HumanAgentBrain that human controls. If another agent was selected, the message is send to that agent (assuming that the "Move to..." option was also available according tot the fill_context_menu method). Then, agents can parse these message as normal and act accordingly. This allows all the described options as far as I can see. In fact, it may also make it easier for use to extend this with helper-functions. For example: We can defined the ActionContextMenuMessage that sends an action name and arguments, which can be automatically parsed by any AgentBrain on receiving such message and relayed as "user request" to the decide_on_action of the receiving agent. We can make other message types to support other helper-functions (e.g. a CallFunctionContextMenuMessage which is parsed behind the scenes where the mentioned function is directly called).

Thus, there are two methods of using this new feature:
o To communicate something to the human agent being controlled (e.g. do a specific action, send a message, or something else)
o To request something from another agent, on behalf of the human agent being controlled (e.g. can you do action X, send a message, or do something else).

2. Good summary of the two interactions we want to offer. To help facilitate ease-of-use, it may be convenient to seperate these two in different 'overrideable' functions in an AgentBrain / HumanAgentBrain?

We should be able to use the context menu for a variety of things. A number of options that the context menu should be able to display:
o Do action X on this on object (or request that another agent does this)
o Send a message (to a specific agent on which the menu was opened)
o Do a meta-action: an action that consists of multiple sub actions. E.g. move to this location, a command that consists of a sequence of Move actions.
o Other custom things. E.g. execute a specific function in the human agent when that option is clicked.

3. To readily support these all sounds like quite a lot. Perhaps first focus on the second option in this list? Since the parsing of a message might achieve all other options. Also, the idea of "meta-actions" is something I plan on implementing someday, so let's atleast wait with that for the moment.

o Properties or type of the object on which was right clicked. E.g. A ball object might have the PickUp action as a option, a wall doesn't.
o Properties of the gridworld. E.g. a certain event occurs at tick 10, which prevents anyone from doing certain things.
o Properties of the human agent. E.g. a human agent that doesn't have wheels or legs cannot do the move action, thus that is not an option.

4. You listed these three as conditions on which the context menu may depend. I agree with them and understand where they are coming from. But may it no be already a bit much? Why not first try and focus on the first bullet? For supporting the second one need to receive a GridWorld instance / world properties in some functions that need to be implemented by a user or perhaps an event system that may block certain context menu options. The third bullet (also for the regular agents) seems to be automatically supported if the functions that do the work are located in agents (which can read their own properties).

Permissions, what is the human agent allowed to task/say/communicate to the robot. E.g. the robot might have an action that only certain people can command the robot to do.

5. Similar as in 4; sounds like something we can add later, or is their already a need for this in a project?

fill_context_menu(self, requester_agent_ID, selected_object_ID, selected_location)

6. I don't really understand the signature of this method and how it relates to the variables mentioned in the text. Is the "requester_agent_id" the ID of the Human Agent in whose view the context menu was opened and clicked? Also, the naming implies for me that their is already a request made for the agent do something. But if I understand correctly, the "requester" and "selected" regard the actual click event right? So perhaps rename this signature to create_context_menu(self, clicked_agent_id, clicked_object_id, click_location) (or something similar, as clicked_agent_id still sounds a bit confusing, perhaps agent_id_who_clicked is better)?

The API receives the request with these parameters, and calls the fill_context_menu function from the desired agent.

7. Who is the "desired agent" here? The Human Agent who 'performed' the click? Or the agent that was selected while a Human Agent performed the click? I think this caused a lot of confusion for me earlier when we were discussing this feature, and I am still unsure which agent actually does this. For example; if the Human Agent selectes an Agent and then right-clicks on an object, is it the Human Agent that decides what the context menu is or is the Agent? In the former we follow the idea that the Human Agent defines what it can send to other agents (which may or may not listen to it), in the latter we follow the idea that each agent decides for itself what others can command it to do.

When the user now clicks on one of the options, an new userinput API request is sent, with as data the optionType and optionSpecificData.
This data is received by the human agent as with any other userinput: in the decide_on_action function in the userinput variable.

8. This seems to imply that the Human Agent is the one who decides what the context menu will look like (see 7)? In addition, a click on a context menu is always first send to the Human Agent who 'performed' the click? For example, when a user opened a menu and clicked the "Go here" option while an Agent was selected, the HumanAgentBrain sees its user_input variable changed containing this click action and may or may not decide to do something with it (e.g. send a MATRX message to that agent to move there). Do I understand this correctly?

Action: When the user clicks on this. The agent should perform a MATRX action.

9. Great suggestion to simplify things! I would suggest going a step further and wrap this all in a function to the HumanAgentBrain, e.g.: add_action_context_item(self, menu_text, action_class_name, action_args). Or yet another step further: create such a method for each action we have, e.g. add_context_move(self, menu_text), add_context_pickup(self, menu_text, selected_obj, permitted_range), etc.

Message: Clicking on such an option makes the current human agent send a specific message. E.g. a global message that an object has been found, or sending a message to another agent.

10. See my general remark 1. It seems to me that with a Message you can achieve almost anything already (including tasking your own HumanAgent if you send the message to the HumanAgent itself). In this case the selected_obj_id is the receiver, and if none is selected the HumanAgent itself is the receiver.

Custom: for all other things that don't fit in the above categories.
E.g. a meta-action or sending custom messages.

11. Again, I should wait with the meta-action idea for now. Especially because you can achieve it by sending a message to an agent (e.g. "Go find red blocks") which translates this into a certain behaviour. Later, when we devised a more sophisticated way of programming an AgentBrain using high-level tasks/goals/meta-actions, we can think about supporting this more easily. In which case we could use the same as the "Action" suggestion, but instead of sending Action.__name__ we would send something like Behaviour.__name__, or whatever we call our meta-actions.

If a context menu option type Custom is clicked, this can be caught by the user in the decide_on_action as well. But they can decide how to further process this themselves.

12. This sounds a bit to dicey for me. If I understand correctly the user_input variable in such a case would contain the name of the function that has to be performed. This function, with a bit of Python magic, can be called by a MATRX user in its custom HumanAgentBrain (since each Python class object is a dict with as keys variables and methods and as values the respective variable and method). But I suggest we simplify this as well for them and allow the API to find the function with that name in the relevant AgentBrain and call it as a callback. Again, I would pause this for the moment. The function is already quite complex :P

This solution provides a context menu that is easy to use for beginners, but with enough options for more advanced uses.

13. Following the MATRX Philosophy (which I make up as we go), "easy to use" is defined as doing something with very few lines of code, that can be done in only one way, and this way is documented and has a tutorial. It is good to check if this is all the case when it is implemented, perhaps use some people as test-subject :D

For example, if a user wants a context menu with all actions the agent can do, it's as simple as:

14. This sounds as very nice first version of this feature. Not meaning that we can't build the full feature for the release V1.1, but that it allows us to get a feel of it before we implement the rest. It also has the advantage that it requires all things to be implemented that can be later extended to support the other functionalities.

The code above can be easily extended to add checks on which actions specifically are possible for an object, or adding the code above to an agent such that it can be tasked by the human agent.

15. Final remark; If a HumanAgentBrain contains the code on what should be in the context menu (as well as how to handle the click) on an arbitrary click event with an arbitrary selected object, and arbitrary location, the decision what should be in that menu and how to handle a click can by definition only be based on the Human Agent's state. It cannot be based on things that are outside this state. At the moment I cannot think of a case where this would cause a problem, but perhaps something to think about.

[thaije]

Hey Jasper, thanks for the feedback!

Overall I agree with your points. To me it also sounds like a good idea to make all the context menu options work via messages. Same for seperating the two methods (targetting human agent, or another selected agent) for using the context menu in different 'overrideable' functions in an AgentBrain / HumanAgentBrain is also a good idea.

Some other remarks to clarify your confusion.

First confusion comes from what is going to be implemented. The functionality I named in (your) point 3, 4, and 5 were mostly meant as: our code for the context-menu's should not make these things impossible. Not meant as: our code facilitates all these things or is something we should implement. I was not planning on implementing any meta-actions or permissions logic whatsoever, but if a user wanted to do so, our code should not make it impossible to combine with the context menu code.
With our messages approach as you described I think all types should be possible (which is sufficient in my opinion) :)
For point 4 I think I wrongly worded "properties of the gridworld". I also had in mind for it to be sufficient if you have access to the "state", as that includes contains info on the gridworld and objects, and the agent properties are ofcourse accessible from within the agent brain.

As for point 6, 7 and 8, you are right. The passed parameters relate to the click event for opening the context menu. However, if we are going to split the functionality as you suggested, it could be come something like this:

create_context_menu_for_self(self, clicked_object_id, click_location)

create_context_menu_for_other(self, agent_id_who_clicked, clicked_object_id, click_location).

Agents would then only have the bottom one. Human agents has the top one (or maybe both? They have both because of inheritence, but can a human agent task another human agent with the context menu?). So for the case where the human agent opens a context menu on an object, the API calls the top function above in that human agent's menu. If the human agent first selects another agent, and then opens the context menu somewhere, the API directly calls the create_context_menu_for_other from the selected agent. The reason for doing it this way, is that a human agent doesn't know all capabilities etc of the other agent who it wants to ask.

However, I think you found a conceptual issue with this. Because this method above implies that the human agent asks the other agent: "Hey, What can I make you do? Can you give a list?". In our code this is done directly by the API without going through the human agent. The agent then passes this list via the API to the frontend. And then you can choose from those options.
An alternative would be that you never leave the HumanAgentBrain of the human agent you're controlling. So it becomes a list of things which you think the other agent can do, which might not be the case.

I don't know which one is better, or more realisting, but I do think the second option would make it harder to use, as you need to program some method of knowing what other agents can do in your human agent.

As for your other points, if we switch to using messages for everything, most of the other points will be solved I think. In addition to using messages we can then implement some functions especially for actions, to make those easier to use. Such as the add_action_context_item functionality which you describe.

I'll make a start at this tomorrow, but I do need it for co-active teaming which uses custom message types. So I might need to add some (early version) code that makes that possible as well.