Extended Master's thesis project with the goal of developing a complete machine vision system for the UR10e-based automated biological lab. A combination of competent hardware and a robust software stack provide a solid basis for visual quality control for the lab.
This project is designed to run on a Raspberry Pi 4, a device based on the ARM architecture. This means some adjustments are required before porting "regular" x86 code to the platform.
Most importantly, pyrealsense2, the python wrapper for the RealSense camera SDK needs to be built from source for ARM. All python packages are installed globally on the RPi instead of using a virtual environment.
All other dependencies are specified in the requirements.txt-file, and can be installed by
pip install --requirement=requirements.txt
The Flask server responsible for exposing internal methods is started by running
python server.py
A startup script on the RPi takes care of this whenever the device reboots. This means the web server should always be running on adress 10.10.0.254:5000.
Once the service is running, camera requests can be made to the same IP declaring intention with a different endpoint:
Barcode recognition and decoding is done by sending a request to
10.10.0.200:5000/read_barcode
The endpoint returns the decoded information from the most central barcode in the cameras view, along with the particular barcode type.
The request can be modified to perform several image captures and computations, increasing the probability of reading the sought-after barcode correctly. This is done by adding another parameter:
10.10.0.200:5000/read_barcode/<t>
Increasing t increases the computation time since the request is repeated t number of times before returning a result.
To see what the vision system is currently viewing, visit the root index
10.10.0.200:5000/
or
10.10.0.200:5000/livefeed
in any browser. The system will capture 8 frames per second and update the content of the html continuously.
The development of this project is split into 7 stages:
- Research and notes
- Pose determination
- RealSense camera
- Reading barcodes
- Generating barcodes
- Image processing
- Flask API programming
Content and code from each stage is kept in its own directory, archiving each component by itself before combining into the final production/ code base.
The vision system is based around the Intel D435i Depth Camera, a competent and developer friendly camera with lots of features for future expansion. At this point, only its basic 1080p RGB camera sensor is used to capture images for processing.
Controlling the camera and processing the captured image data is a Raspberry Pi 4 Model B 4GB. This microcomputer is powerful enough to perform image processing and hosting a small web server through which quality control data can be communicated to external systems.
The camera and RPi are mounted together in a case, attached to the gripper of the UR10e robot arm with a 3D-printed mount. The 5V power required by the RPi is supplied by splicing the gripper power cable and separating its 24V and ground leads, after which the voltage is stepped down using a car charger/buck converter.
This project relies on
- OpenCV and its Python bindings for image processing.
- NumPy for its data and image manipulation.
- librealsense and more specifically pyrealsense2 for interfacing with the RealSense camera.
- treepoem and pyzbar for generating and decoding barcodes, respectively.
- Flask is used to host a web server, exposing system methods for remote usage.
- Intel D435i Depth Camera
- Raspberry Pi 4B
- USB-C => USB-A 5Gbps cable
- 3D printed mount
- M8 cable extension for power (Anders)
- 3A car charger (Produktwebben) (Anders)
- USB-C => USB-C power cable (Rikard)
- Network (Anders)