Adding a New Block to the Lab

In this text I describe what has to be done to add a complete new block to the open-roberta lab. This is not a complex task, but a lot of work. If you are an open source enthusiast and have ideas how to simplify or automate this, mail to reinhard.budde AT iais.fraunhofer.de.

The work is divided into wo parts:

• declare the block using blockly ("front-end part")
• implement the AST class and implement visiting this class to the worker/visitor architecture ("back-end part")

As an example I will add the block "robActions_aifes_newneuralnet" (new neural network), which b.t.w. is part of a neural network extension of the lab.

Note, that I'm using the git bash (because I've installed git on my windows machine and like the bash).

Modifying blocky - overview

I have to generate the blockly resources for the lab and to do this I have to

• clone the blockly git https://github.com/OpenRoberta/blockly.git
• on linux: follow the "install on linux" part of the README of the git repo (I need python 2.7 :-< and the closure compiler)
• on linux or windows: install docker (easy, follow the instructions in the net) and do everything in a container without further installations.

I prefer docker and on windows I cloned blockly into D:\git\blockly. To generate the blockly_compressed.js file and the msg directory I have to run

docker run --mount type=bind,source=/D/git/blockly,destination=/opt/blockly/blockly openroberta/blocklybuilder:1.0.0

How are blocks added? Blocks are either sensors, actors or common blocks. Some are used by many plugins, some are very specific. Sensors are added in almost all cases in the robSensorDefinitions.js file. These sensors have no XML-sub-elements and return a single sensor value. When adding the sensors of the nano33blesense, I designed more complex sensors: the blocks return a boolean value and have at least one XML-sub-element. The boolean value tells, whether data from the sensor is available or not, the XML-sub-elements must contain a variable, into which the sensor value is stored (if data available is true). Sensors of this kind are declared in robSensorDefinitionsDataAvailable.js. All other blocks are defined in other ``*.js' files, most of them in `robActions.js'.

Workflow:

• cd /D/git/blockly
• git checkout -b feature/addNewNN
• I have to look for a file, which contains block definitions matching your robot and contains similiar blocks, and, as usual, look for the most similiar block, copy and modify. If I detect too much duplications of code, I should design a better architecture, but this is not simple.
• The next step is testing the new block. In the directory tests there are many html-files. Open an appripriate one (for the new block ``newNNI useprogramConfiguration_playground_nano33ble.html`). It is ok to copy one of the files and modify it for my needs.
• I designed the block for robActions_aifes_newneuralnet as

Blockly.Blocks['robActions_aifes_newneuralnet'] = {
	/**
	 * connects to the AIfES library
	 */
	init : function() {
		this.setColour("#646464");
		this.appendDummyInput().appendField("new network");
		this.appendValueInput("NUMBEROFCLASSES").appendField("number of classes");
		this.setPreviousStatement(true);
		this.setNextStatement(true);
		this.setTooltip("create a new neural network classifier for the number of classes given");
	}
}

• sometimes I rebase on master (conflicts are rare, because adding blocks is not a frequent task): git fetch;git rebase master
• if I have finished the work, I must rebase and push:

git fetch;git rebase master;git checkout master;git merge feature/addNewNN;git push

• I run the docker container as described above.
• I copy the blockly_compressed.js file and the msg directory into my feature branch of the openroberta-lab git
• later I delete my feature branch git branch -d feature/addNewNN

Making the block available in the backend

At any time a robot plugin is acivated. If I activate the nano33ble plugin, for instance, the property file nan33ble.properties is used. it defines a lot of stuff needed in the lab, among others all the resources mentioned below are declared in this file. I always check the robot property file of the robot I want to modify.

the *.yml block declaratin

The blockly XML is sent to the server and transformed to an abstract syntax tree (AST), which is more appropriate for checking and code generation. This is done with JAXB embedded in a small framework designed by us. Thus, I have to declare the new block for this framework. This is done in a YAML file. There is one global robotCommon.yml in the OpenRobertaRobot project, but the concrete robots have to add their robot-specific blocks. My block belongs to the arduino group and thus must be added to arduino.yml in the RobotArdu project. The type array must contain the name from blockly:

NEURAL_NETWORK_NEW:
    category: STMT
    implementor: de.fhg.iais.roberta.syntax.neuralnetwork.NeuralNetworkNew
    type: [robActions_aifes_newneuralnet]

the toolboxes

User draw the blocks out of toolboxes. These toolboxes are defined in the files program.toolbox.beginner.xml and ´program.toolbox.expert.xml´ of the directory nano33blesense. I added the new block to the expert toolbox only:

<category name="TOOLBOX_NEURAL_NETWORK" svg="true">
        ...
        <block type="robActions_aifes_newneuralnet"></block>
        ...
    </category>

the implementor class of a block

I have to implement the AST class NeuralNetworkNew, which corresponds to the blockly block:

• the method public static <V> Phrase<V> jaxbToAst(Block block, AbstractJaxb2Ast<V> helper) is needed to convert a jaxb-xml-representation to an object of the class NeuralNetworkNew.
• add a Java-field for each xml-field and xml-value (in my case numberOfClasses) together with a getter (no setter!)
• the method public Block astToBlock() is needed to transform this object to a jaxb-xml-representation (before it is returned to the frontend). This is needed for the case, that annotations (errors/warnings) are added to the AST, which must be displayed by the fronend.

The method, which accepts one of the many visitors of the lab, has consequences at many placed in the visitor framework of the lab und is the most interesting one from a software engineering point of view. The visitor framework has been redesigned in 2018 and needs a further redesign, so be careful! If you are not familiar with the visitor pattern, there are a lot of good resources in the net and of course there is the GOF book Design Patterns.

    @Override
    protected V acceptImpl(IVisitor<V> visitor) {
        return ((IArduinoVisitor<V>) visitor).visitNeuralNetworkNew(this);
    }

The idea is the following and will be explained in the context of the nano33blesense and the block robActions_aifes_newneuralnet and the corresponding AST class NeuralNetworkNew:

• we have an AST, that represents the program which was written using blockly as concrete syntax of NEPO.
• this AST has to be visited for many purposes, e.g. by the
  • ArduinoUsedHardwareCollectorVisitor to collect all sensors and actors used in the AST
  • ArduinoBrickValidatorVisitor to check the consistency between program and configuration
  • Nano33bleCppVisitor to generate code for the C/C++ compiler.
• to express, that all visitors have to visit the NeuralNetworkNew object, I added the following method to the top level public interface IArduinoVisitor<V> of all arduino-related visitors:

    default V visitNeuralNetworkNew(NeuralNetworkNew<V> nn) {
        throw new DbcException("Not supported!");
    }

• if another member of the arduino group, which doesn't support this block, would use the block, an exception is thrown. All visitors used by the nano33blesense (the 3 mentioned above) have to override the default implementation.
• to see all the places, which I have to override when I add a block, I select a similiar block and search for all the occurences of the visit:NameOfTheAstClass: method
• to avoid code duplication as much as possible the visitors have a inheritance hierarchy to ease re-use for other robots. Due to an imperfect implementation on our side (:-<) this is not always as elegant as possible. You are invited to help!

I added the following code to the visitors:

• ArduinoUsedHardwareCollectorVisitor: hardware may be used in the number of classes expression, thus the expression must be visited:

    @Override
    public Void visitNeuralNetworkNew(NeuralNetworkNew<Void> nn) {
        nn.getNumberOfClasses().accept(this);
        return null;
    }

• ArduinoBrickValidatorVisitor: the same as immediately above
• Nano33bleCppVisitor adds the code for this block to the StringBuilder sb. Here a comment is added for debugging. See the real code in the lab git repo.

    @Override
    public Void visitNeuralNetworkNew(NeuralNetworkNew<Void> nn) {
        this.sb.append("// visitNeuralNetworkNew");
        return null;
    }

After this has been done, I started a local server and

• selected the nano33ble
• switched to the expert toolbox
• draw the block for a new neural network and connected it to the start blockly
• pressed the <> button to see the code generated. It shows:

// This file is automatically generated by the Open Roberta Lab.

#define _ARDUINO_STL_NOT_NEEDED
#include <Arduino.h>

#include <Arduino_HTS221.h>
#include <Arduino_LSM9DS1.h>
#include <Arduino_LPS22HB.h>
#include <Arduino_APDS9960.h>
#include <NEPODefs.h>


float xAsFloat, yAsFloat, zAsFloat;
int _led_L = LED_BUILTIN;
int rAsInt, gAsInt, bAsInt;
void setup()
{
    HTS.begin();
    IMU.begin();
    pinMode(_led_L, OUTPUT);
    BARO.begin();
    APDS.begin();
}

void loop()
{
    // visitNeuralNetworkNew
}

Now I have to add unit and integration tests. I can follow the style of similiar classes. Then I rebase my feature branch, commit my work and push my feature branch or create a pull request.

Adding the block is finished :-).