Execution engine for robotics applications inside RADI

[jmplaza]

@dmariaa and me are thinking about introducing more robustness in the execution of the exercises inside RoboticsAcademy. Currently there are FIVE connections between the RADI and the browser:

• (1) WebServer, https connection.
• (2) VNCserver. Includes (2a) Gzviewer and (2b) console.
• (3) Manager.py, through websockets
• (4) Exercise.py, code-websocket. For sending robot brain source code from the browser to the RADI, and sending "load-in-robot", "pause", "run", etc. commands.
• (5) Exercise.py, GUI-websocket. For sending internal robot brain data from the RADI to the browser in order to be displayed at the webpage of the exercise. For instance processed images, or virtual forces, etc..

The WebServer, the VNCServer and the manager.py run inside the RADI. Manager itself starts on demand the corresponding exercise.py for each specific exercise when selected by the user, and it is also run inside the RADI.

In order to simplify the frontend development at the browser and to simplify the communications (3)(4) and (5) may be merged into a new (3'), a single websocket connection between the RADI and the browser. It should hold all the messages currently carried by (3)(4)(5). With this refactoring there would be THREE connections between the RADI and the browser: (1)(2) and (3').

It would be also good to standardize the message protocol for that websocket connection between the Manager.py and the browser. This way the implementation of the communications would be more clear and easier to develop or extend, both in the browser and in the Manager.py itself.

What do you think?

[jmplaza]

Regarding this Robotics-Academy-Execution-Engine (RAEE) I see three levels for any robotics application:
(a) the Scenario, mainly a simulated world in Gazebo.
(b) the robot Body, its sensors and actuators. They are both active software agents, they publish/receive ROS topics
(c) the robot Brain,
Let's call this the BBS paradigm = Brain + Body + Scenario.

Beyond execution of the robotics application, RoboticsAcademy also requires visualization capabilities (GUI), in order to help the user to debug the application and provide a better UX.

Let's call RAEE Protocol to the set of messages to manage the execution of a robotics application in RoboticsAcademy.

Regarding the Scenario and the robot Body I see several commands in the RAEE Protocol:

1. connect and disconnect: to start and finish the communication between the Manager.py and the browser
2. launch : to start the Scenario of a given exercise and the robot Body. Manager should start the Gazebo simulation with the corresponding world file including both the file and the sensor and actuator plugins. In the case of physical robots, Manager should start the sensor and actuator ROS nodes (drivers). The configuration file maybe a parameter of this message. This message is very important, it starts the context of the robot application.
3. reset: to restart the Gazebo simulation from the initial point again
4. terminate: to kill the sensor and actuator drivers/plugins and the Gazebo simulation itself

Regarding the robot Brain I see several commands in the RAEE Protocol:

1. load in robot: to load a source code into the robot brain (without executing it yet), maybe replacing the previous one or just from scratch. The source code to be loaded is a parameter of this message
2. run: to execute the robot Brain already loaded into the robot. In case of no Brain loaded yet this operation is idle.
3. stop: to permanently cease the execution of the current robot brain. It will forget any internal state. In case of a following execution it will start from the initial state.
4. pause: to cease the execution of the current robot brain. It will keep any internal state when resumed.
5. resume: to go on with the execution of a paused brain.

Regarding the visualization (GUI) I see several commands in the RAEE Protocol, mainly to show standardized data (vectors, images, pointclouds), etc...

[Apoorvgarg-creator]

In my humble opinion, I think it would increase the performance and user experience.I would suggest the use of socket.io. Compatible with REACT also socket.io with react hooks

[jmplaza]

Sounds great @Apoorvgarg-creator !

Yes, there are two sides to implement the RAEE Protocol: the browser side (JS), and the RADI side (in Manager.py, in Python). For the browser side definitively socket.io library is a good choice (it is widespread, compatible with REACT...)

For the Manager side, there is Python implementation of socket.io too: https://github.com/miguelgrinberg/python-socketio

[dmariaa]

In my humble opinion, I think it would increase the performance and user experience.I would suggest the use of socket.io. Compatible with REACT also socket.io with react hooks

The main problem I see with socket.io is that it is mandatory to use it's server, node.js based... our backend/server is django based, and though there are some efforts to join both technologies (https://django-socketio.readthedocs.io/en/latest/), they are outdated working with old versions of socket.io only...

[dmariaa]

Sounds great @Apoorvgarg-creator !

Yes, there are two sides to implement the RAEE Protocol: the browser side (JS), and the RADI side (in Manager.py, in Python). For the browser side definitively socket.io library is a good choice (it is widespread, compatible with REACT...)

For the Manager side, there is Python implementation of socket.io too: https://github.com/miguelgrinberg/python-socketio

Sorry @jmplaza I didn't read your answer before writing, I was a bit outdated about python-socket.io integration... Anyway I still think there are some reasons to avoid adopting socket.io at least right now... our client-server communication it's not very aggressive, it's only one-to-one as one and only one exercise client connects to one RADI server, so I still think the benefits of adopting socket.io still don't worth it...

[dmariaa]

As far as I see, the exercise cycle can be modeled by this state machine, for all the exercises we have...

[dmariaa]

Server side the exercise manager should follow this scheme... actions that trigger state changes can be delegated to both generic handlers or been overriden by exercise specific handlers if needed...

[dmariaa]

Client side, the exercise manager can keep the state of the exercise, so UI widgets should have only local state... client side exercise manager can follow an interface like this...

[dmariaa]

So the main challenge should be only to keep state synchronized between server-client exercise managers...

[dmariaa]

This is how I see it, I think the basics of this architecture would be simple, easy to develop and maintain and to extend if needed... of course this is only the core of communicacion and the very basis of the server side engine...

Server side handlers for actions and states would be the real code of the exercises, we need to find a simple, easy and robust way to build them so they can be easily overriden if needed... In my mind something like ECS (entity component system) patterns used frequently in game development and in most game engines like Unity of Ureal can be a good starting point.

[dmariaa]

In next days I am updating the new architecture branch with a small prototype of RAM (Robotics Applications Manager) with launcher functionality that follows this idea, trying to follow SOLID principles as much as possible. Any ideas are welcome.

[dmariaa]

The manager, when launch transition is invoked, will instantiate a Launcher that according to a JSON configuration depending on the exercise will know what modules/programs/commands to invoke on the server. A sample of the JSON configuration for launcher is shown below:

{
  "template": "react",
  "launch": {
    "0": {
      "type": "module",
      "module": "gazebo",
      "parameters": [],
      "world_file": "sample_world_file.world"
    },
    "1": {
      "type": "command",
      "command": "mkdir /tmp/output_directory"
    }
  }
}

[dmariaa]

We have a new class diagram that includes the actual situation of the new manager (RAM - Robotics Application Manager)...

[dmariaa]

The new configuration includes the application to run...

{
  "application": {
    "type": "python",
    "entry_point": "$EXERCISE_FOLDER/web-template/entry_point/exercise.py"
  },
  "launch": {
    "0": {
      "type": "module",
      "module": "ros_api",
      "resource_folders": [
        "$EXERCISE_FOLDER/web-template/launch"
      ],
      "model_folders": [
        "$CUSTOM_ROBOTS_FOLDER/f1/models"
      ],
      "plugin_folders": [
      ],
      "parameters": [],
      "launch_file": "$EXERCISE_FOLDER/web-template/launch/simple_line_follower_ros_headless_${circuit}.launch"
    }
  }
}

[dmariaa]

Which is modeled by the IRoboticsPythonApplication interface

[dmariaa]

This is the state machine after our last meeting to discuss this ideas.