youtube Link: https://www.youtube.com/watch?v=Ian9UkIapt0
We propose a novel software architecture and system design for remotely operating industrial manipulators using a virtual reality interface. The system is based on reconstructing the 3D environment around the manipulator, transmitting and displaying it to the operator in a virtual reality headset via the internet. The operator can then use motion trackers and other input devices to control the manipulator and interact with the environment. The proposed framework leverages and integrates a variety of technologies across multi-computing platforms, such as Unreal Engine 4, MQTT, Nodejs, etc. In this paper, we display the design and implementation of this system, as well as the results of tests conducted to evaluate its performance and effectiveness. Overall, the software framework is shown to be a promising approach for improving the efficiency of developing a VR-based industrial manipulator teleoperation system.
As shown in Figure at the top, the operator is using a motion tracker to interact with a VR workspace that is being transmitted from a factory site. The virtual workspace is a 3D representation of the actual workspace where the manipulators are located, and it is captured by a 3D camera. The operator can move around within the VR workspace and interact with the objects as if they were physically present at the factory site. By using input devices such as motion trackers, joysticks, and buttons, the operator is allowed to remotely controls the manipulators at the factory site to perform tasks from a distance.
As shown in Figure above. Two x86-based computers were set up in different physical locations and connected to the same network. The teleoperator runs a VR application with a headset device on one of the two computers. The other computer runs Linux and all microservices that orchestrate the manipulator and depth sensor.
- The operator first initiated the request to scan the remote workspace and once it was completed, a point cloud file was downloaded and rendered in the VR workspace. This point cloud file is stored and can be used in the future without rescanning.
- The operator uses a hand controller to record the manipulator tracks as well as additional information, such as spray gun state, in a constant recording rate
- Once it is completed, the record track data was sent to the dedicated Linux service and handled by VR base task manager service.
- The VR-based task manager will filter and further request path planning with robot kinematics service. The trajectory file was saved and ready to playback.
- The actual playback on the manipulator can be triggered by either a remote teleoperator through a VR application or an onsite operator via the web frontend application.
As shown in Figure above, The system supports multiple other workflows ranging from debugging and hyperparameter tuning via the web frontend. Operators of the manipulator can tune the lower-level motion control parameters such as the PID coefficient. sensor reading, command refreshing rate as well as digital model twisting such as joint angle allow range. joint angle max speed. In addition, technicians can also simulate and watch the real-time manipulator pose or trajectory playback result through a web interface.
MQTT is a publish/subscribe protocol that is widely used in IoT development and edge computing due to its minimal data schema, easy-to-understand message process flow, and ability to run on a variety of devices and programming languages. It is well-suited for handling sensor data ingestion and consumption and offers a range of transport protocols and message retention policies.
HTTP is the most widely used protocol for exchanging information when a response is required for a request. It involves a client initiating a request and a server processing and generating a response. HTTP can be easily extended and deployed for a variety of purposes, such as triggering actions and editing configurations. HTTP clients are supported by a wide range of programming languages and software, making it a solid choice for function invocation, configuration adjustment, and other CRUD (Create Read Update Delete) operations.
In the messaging protocol MQTT and HTTP, JSON formatted data is used for better compatibility and debugging speed.
Node.js is a runtime environment that allows developers to execute JavaScript code on the server side, rather than in a web browser. It is based on the Chrome V8 engine, which is a high-performance JavaScript engine developed by Google. Node.js is designed to be lightweight and efficient and is often used for building scalable network applications.
One of the key features of Node.js is its event-driven, non-blocking I/O model. This means that Node.js is well-suited for input/output (I/O) intensive tasks, such as reading and writing to databases or interacting with APIs over the network. Node.js can handle many concurrent connections with low overhead, making it a good choice for applications that need to process large amounts of data or handle a high volume of requests.
Python is a popular, high-level programming language that is widely used in numerical computation and statistical analysis. One of the reasons that Python is well-suited for such tasks is the abundance of specialized packages and libraries available in its ecosystem. These packages and libraries are highly optimized.
In the final implementation of the system, both Node. js-based JavaScript and Python were used in order to take advantage of the strengths of both languages. Node.js was used for tasks that required high-performance I/O processing such as HTTP web service, and file read-write, while Python was used for tasks that required more computational power such as robot kinematics and point cloud processing.
Figure above shows the proposed architecture, which is based on the concept of microservices, which involves breaking down a monolithic application into smaller, highly cohesive, and loosely coupled modules. This allows for the use of multi-threaded computing, faster prototyping and testing, and the ability to easily swap out individual components.
This architecture is designed to have minimal programming requirements, meaning that each component can be implemented in the programming language of the developer's choice without the need to consider the design of other components. This allows for greater flexibility and modularity, as developers can focus on implementing their specific components without worrying about compatibility with other parts of the system. As a result, this architecture allows for faster development and easier maintenance, as changes to one component can be made without affecting the rest of the system.
The proposed architecture uses HTTP for function calls and MQTT for data broadcasting as its communication protocols. This allows the microservices to run in a wide range of environments and network configurations, as both HTTP and MQTT are widely used and supported. By using these protocols, the microservices can communicate with each other and perform their intended functions without being restricted by the specific architecture or topology of the system. This helps to ensure that the system is flexible and adaptable and can be easily integrated into different types of environments.
In addition to the microservices architecture, all components in the system are packaged into Docker containers. Containerization technology like Docker allows for the creation of clean runtime environments for each microservice to run in, acting like a sandbox to isolate the microservice from the rest of the system. This helps to ensure that each microservice has the necessary computing resources and can operate without interference from other components. By using containerization, the overall architecture can avoid issues such as resource competition and dependencies conflicts, resulting in improved efficiency and stability.
The core architecture of the system is built on a node.js server application, which serves as the central server responsible for collecting sensor data, dispatching manipulator commands, and acting as a proxy for backend services. This application runs on a dedicated Linux server that is connected to an embedded microcontroller unit and depth sensor, and it provides control interfaces for other services to interact with and exchange data. The node.js runtime model uses an event-based programming approach, in which code is executed based on events rather than in a predetermined sequence. The underlying engine is based on a polling model, which allows for the efficient orchestration of system tasks and data exchange.
To manage the manipulator model, a Python-based microservice was introduced. This microservice can compute the forward and inverse kinematics of the manipulator and generate feasible trajectories for its movement. This service exposes its functionality through an HTTP server and runs in its container, allowing it to operate independently and communicate with other components of the system as needed. The use of a microservice for this specific task allows for greater modularity and flexibility, as the manipulator model can be easily updated or modified without affecting the rest of the system.
To interact with the Azure Kinect depth camera, a Python-based microservice was introduced. This module is responsible for handling HTTP requests and returning point cloud files for other services to process. Since there are multiple depth sensors involved, certain algorithms are used to filter and register the point cloud data to generate a final virtual workspace point cloud. To avoid issues with multiple requests attempting to access the physical device simultaneously, this module is designed as a singleton process, meaning that only one instance can be active at a time. This helps to prevent invalid scanning states caused by duplicate device access.
To provide a user interface for the system, a React frontend application was developed. This application serves as a data visualization viewport and system state management tool, allowing users to view the intermediate states of the system and interact with it through a web browser on any device. The interface also can send commands such as emergency systems to disable, activating pre-defined manipulator poses, updating configurations, and more. In addition to the frontend application, a web-based 3D simulator was developed to simulate the manipulator state using the Three.js and URDF loader packages. This simulator allows the operator to test and visualize the trajectory of the manipulator before running it in a real environment, providing a useful tool for planning and testing.
To address the issue of data mismatches in the calculation of VR manipulator teaching positions, a Python-based microservice was deployed to pre-process data received from VR applications. In particular, this microservice is responsible for handling differences in the coordinate systems used by the VR engine (Unreal) and the mathematical model for manipulators. Unreal uses a left-hand coordinate system with clockwise positive rotation on the Z axis and counter-clockwise positive rotation on the X and Y axes, while the mathematical model uses a right-hand coordinate system with counter-clockwise positive rotation on all axes. This microservice is also responsible for initiating manipulator track playback by making HTTP requests to the core orchestration applications previously mentioned. By handling these tasks, the microservice helps to ensure that the VR and manipulator systems can work together smoothly and accurately.
The VR application was built using blueprint technology in the Unreal Engine. This powerful rendering engine allows for the rapid creation of a demo metaverse environment and easy integration with the dedicated Linux server that runs all the microservices. The use of the Unreal Engine allows for high-quality graphics and a wide range of features, making it an ideal choice for building immersive VR experiences.
