ARRobotinteraction.Base
This component is the center of the Unity side of this framework. It handles the communication (REST & adapted ROS#), URDF-Robot instance management and it uses the ARRobotinteraction.Data & ARRobotinteraction.Sensor namespace to fulfill the user's wishes.
Every scene must have an instance of the RosConnector prefab, which is pre-configured and located in the "ARRobotInteraction.Base/prefabs" folder. This GameObject has multiple Scripts attached to it:
- RosConnector.cs
- ARCommander.cs
- RestCommunicator.cs
- ARStateSubscriber.cs
- MainThreadDispatcher.cs
- DebugPublisher.cs The contents of these files will be described in the following:
This is the default RosConnector from ROS#, which is responsible for connecting to the necessary WebSockets etc.
This is the entry point for the whole application. The main method here is ReceiveMessage(RunningInstance[] runningiInstances). This method accepts an array of type RunningInstance[]. An instance of a RunningInstance basically defines a bunch of variables used by Docker on the Off-Device component. One of them is the instance's id and another one is urdf_dyn, which is this instance's current urdf contend.
The ARCommander manages a Dictionary Dictionary<int, UrdfSyncher> currentSceneDicionary field of current instances. If one of the received instances does not exist, it creates an entry for it with a freshly initialized UrdfSyncher object. After that it calls SynchUrdf(...) on it. The inner working of UrdfSyncher will be described later in this document.
Other than that, the ARCommander also fetches all available components via the REST-API from the OFF-Device, which can then be displayed if the correct public properties are configured.
This is a small custom written REST client for Unity. It uses asynchronous messages with callbacks and interprets answers as JSONS. This class also defines the IP-Adress of the REST-Server.
This class subscribes to the ROS-Topic where the current state of the system is published to, deserializes the string, checks for changes compared to the previous update and if there is any change it sends the new state (runningInstances) to the ARCommander class using the MainThreadDispatcher.
Small utility class which has a queue of Actions to be performed on the main thread. Offers a public function for other threads to deposit tasks, which will be executed in the Update method.
Helper class to make it easier to debug Unit Applications. It subscribes to Unity's Log.Debug(...) & Log.Error(...) methods and publishes the content on a dedicated ros topic.
The ARRobotInteraction.Base namespace has the ability to synchronize a Unity GameObject with a urdf string. This means that it does not only build a GameObject from previously prepared prefabs, but it synchronizes it. This means, that if a new updated urdf string lacks a few components, they will be deleted in the Unity-GameObject as well.
This all is controlled from the UrdfSyncher.cs class.
This class first statically initializes some custom xml tags in the urdf parser Robot class.
This class basically is used non-statically (such that multiple instances can run at the same time) and contains the information to uniquely identify a robot in a scene.
It also creates instances of our custom RobotBuilder class, which does the actual urdf synchronization.
This class handles the actual synchronization of an existing GameObject and Robot instance. (Robot class is only changed minimally from the original ROS# Robot and allows to import custom xml tags more easily).
This class basically does a recursive search through the Robot object - similar to how ROS# does it. The difference is, that this class does not create GameObjects if the GameObject already exists at this location in the tree. Detecting that a GameObject already exists is often challenging, because not all urdf xml tags have unique names.
Thus, this class makes heavy use of the custom utils class, which offers the method GenerateNonReferenceID(object obj), which returns a string, which can be used as an id for the GameObject in question. The nice thing is, that even a changed urdf string would result in the same id at this tree-location, allowing the RobotBuilder to identify already existing objects.
In this step, the RobotBuilder also comes across AttachedDataValues which are our custom xml tags for the ARRobotInteraction.Data namespace and mark the location and content of data-plots. Here the RobotBuilder utilizes a helper class called AttachedDataSynchronizer to command the ARRobotInteraction.Data namespace to generate data plots.
If a certain component does not exist in the urdf anymore but exists as a GameObject in the Unity-Scene, then the GameObject will be deleted. All of these steps happen recursively throughout the whole urdf document tree.
This code is based on the Unity scripts of the original ROS# repository, but was heavily modified to be AOT compilable.
The IL2CPP compiler from Unity, which is needed as a coding backend to build for HoloLens applications, does not support JIT compilation. Hence, all parts where reflection was used, had to be rewritten.
Also, the original ROS# Unity scripts heavily relied on the "Unity.Editor" namespace, which is not usable in-built applications. This means, that the complete GameObject creation had to be rewritten without the use of the System.IO, Unity.Editor and System.Reflection namespaces.