Developer guide

Main apps

These are the main apps in this project:

• Badger: C# GUI app that allows the user to design, run and analyze experiments.
• Herd Agent: C# service/daemon process that is installed in the slave machines to receive job requests, run them and return the results.
• RLSimion: C++ executable that does the actual simulation and Reinforcement Learning process. As input, it receives an XML file with the parameters of the experiment. As output, it generates log files and, if asked to, visualizes the experiment in real time.

The diagram below shows a simplified view of the experiment execution process:

Setting Up Visual Studio

This project is developed using Visual Studio 2017 Community for all the target platforms: Windows (x86/x64) and Linux (x64).

Components

The following components need to be installed from the Visual Studio Installer if you want to compile the projects straight away (in the order they are listed in the installer):

• .NET Framework Targeting Pack 4.5.2 (or either retarget C# projects)
• Git for Windows
• NuGet Package Manager (to automatically download some dependencies)
• VC++ 2017 version 15.9 v14.16 latest v141 tools (or either retarget C++ projects)
• C# and Visual Basic
• Visual C++ for Linux Development
• Visual C++ tools for CMake and Linux
• Windows 10 SDK (10.0.17134.0) (or either retarget C++ projects)
• GitHub Extension for Visual Studio (recommended)
• Microsoft Visual Studio Installer Projects (only required to create the Windows Herd Agent installer)

Windows compilation

All the C# projects and all the C++ projects targeting Windows (those without -linux in the name) should compile straight-forward. Just select the target (Debug/Release and x64/x86), right-click on the solution and click on Build solution.

Linux compilation

Visual Studio compiles Linux sources using a remote Linux machine that receives the source code and returns the generated binaries via SSH. To compile the source code for Linux, the connection to the Linux machine must be added in: Tools->Options->Cross Platform->Connection Manager

We are currently using a Virtual Machine that can be downloaded from here. The connection parameters to c0nnect with this VM are:

• Host name: 192.168.56.101
• Port: 22
• User name: simionzoo (same as the password)
• OS: Ubuntu (x64)

If you want to use your own machine, you need to install the following VSLinux dependencies:
sudo apt-get install openssh-server g++ gdb gdbserver
and OpenGL/FreeGlut-related dependencies:
sudo apt-get install libgl1-mesa-dev libglew-dev libxi-dev

About the source code

If you want to adapt or extend the source code, we recommend you to begin reading the Developer FAQ.

The folder structure of the project is described here and the basic programming rules are given programming rules.

Here you can find how to set directories in new projects and some Visual Studio 2015/2017-related stuff. A mini-tutorial on how to use GitHub from Visual Studio 2015/2017 here.

FAQ

RLSimion

How do I create a new simulation environment (World)?

You must implement a subclass from base class DynamicModel (worlds/world.h) that implements the following methods:

• constructor(ConfigNode* pParameters): used to construct and initialize the world.
  • State variables are defined by calling addStateVariable(NAME,UNITS,MIN_VALUE,MAX_VALUE,CIRCULAR);, where NAME is the name of the state variable, UNITS is the units in which the value of the variable is expressed, the range of values is given by [MIN_VALUE,MAX_VALUE] (Note: use literal values so that the parser can parse them), and CIRCULAR is a boolean that indicates whether it's a circular type of variable, such as angles (false if not given).
  • Action variables are similarly defined calling addActionVariable(NAME,UNITS,MIN_VALUE,MAX_VALUE);
  • Parameter templates in parameters.h can be used so that they can be configured from Badger at design time.
  • To inform the code parser that this class implements a world, the following macro should be included: METADATA("World", WORLD-NAME);, where WORLD-NAME is the name given to the world (shown in the GUI).
• void reset(State *s);: called at the beginning of an episode to initialize the world. State variables in s can be randomly initialized or reset to some initial state of the system.
• void executeAction(State *s, const Action *a, double dt);: called each time-step to calculate the next state of the simulated environment. s is the current state and a the action currently being executed, and this method must update s with the resulting state, dt seconds after the current.

You also need to add the new subclass in the factory method DynamicWorld::getInstance (more about this here).

We recommend you take one of the existing worlds (i.e. worlds/underwater-vehicle.cpp) as a reference.

How do I create a new agent/controller (Simion)?

You must implement a subclass of Simion (simion.h) that implements the following abstract methods:

• double selectAction(const State* s, Action* a);. This is called every time-step so that the agent selects an action to execute. s is the current state and the output action should be set to a (the agent may give value only to a subset of the action variables). This method should return the probability by which the action was selected (currently ignored).
• double update(const State* s, const Action* a, const State* s_p, double r, double probability);. This method is called to let the agent learn from an experience tuple <s,a,s_p,r> (probability should be ignored). It might be called more than once per time-step if Experience-Replay is being used.

You also need to add the new subclass in the factory method Simion::getInstance (more about this here).

Linux/Windows compilation

Since there is a different Visual Studio project for each target platform, new source files should be added to both projects to get the same features in both platforms.