Develop and test Advanced Driver Assistance Systems (ADAS) with the robotcar using Python, ROS, OpenCV and TensorFlow
=====================
Project of the Faculty of Computer Science of the Furtwangen University inside the master degree Mobile Systems.
Part of the master thesis "Entwicklung einer erweiterbaren Simulationsplattform für Fahrerassistenzfunktionen am Beispiel von Python, ROS, OpenCV und TensorFlow"
Developer: Michael Christian Dörflinger
Supervisiors:
The development of driver assistance functions is extensive and expensive. Virtual simulations significantly improve the development process, but do not reflect the reality of vehicle actuators and sensors. Development under physically real conditions is therefore indispensable. The thesis will investigate how such a simulation platform can be implemented using a small robot car, taking into account the expandability of new sensor technology and driver assistance functions.
=====================
(Not the actual used components because you can get it cheaper.)
At the beginning the chassis is simply assembled according to building instructions. Here you can find the install instructions and install videos. Then solder cables for plus and minus on the motor. To use the toggle switch, it makes sense to solder the cables from the battery before they go to the motor driver. The following picture shows how the cables are connected to the motor driver.
From the battery or motor driver, the cables go next to the Step Down Converter.
The Step Down Converter reduces the voltage for the pegboard and the Raspberry Pi. This allows the motor to be supplied independently from other consumers and always receives sufficient power. The Raspberry Pi is connected to the Step Down Converter via a USB-C cable. The plug-in board uses a USB stranded cable, where male plugs are soldered to the jumper cable.
Please note that a ribbon cable of the Raspberry Pi has already been plugged into the CSI slot. In addition, the cooler has been attached and the stacking headers. The port doubler was connected via spacers and the Sense HAT was attached to it. This means that not the pins of the RPi are used but the pins of the port-doubler.
The USB slots of the Raspberry Pi are occupied as follows:
For the GPS module this means the following cabling:
At Raspberry Pi, the I2C bus is extended via an I2C hub as follows:
For the servo controller this means the following cabling:
The rest is connected via the breadboard:
The MCP3008 located on the breadboard is assigned as follows:
It uses the SPI bus of the Raspberry Pi
A more detailed diagram for the robot car wiring can be found in Roboterauto.vpd
=====================
At first download and flash Debian Buster on a Micro SD Card.
After that we will start our Raspberry Pi and open the terminal. A good start is at first:
$ sudo apt-get update -y && apt-get upgrade -y
$ sudo rpi-updateWe begin with changing the interface options:
$ sudo raspi-configThen select 5 Interfacing Options and press enter. There you have to go to different submenus. As example P1 Camera. You also have to press enter. For enabling you have to select Yes and press enter.
You have to enable following interfacing options:
- P1 Camera
- P2 SSH
- P3 VNC
- P4 SPI
- P5 I2C
- P6 Serial
After that we can use the RPi via SSH or VNC headless. You can check the ip address with ifconfig. Then you can reboot with sudo reboot if you want and log in via SSH.
You can connect from a linux terminal to headless started Raspberry Pi via ssh<username>@ip_address then you have to enter the password and press enter.
If you want to connect via VNC you have install as example the Real VNC Viewer on your computer. On the Raspberry Pi you have o change the /boot/config.txt file as example with sudo nano /boot/config.txt. Inside this file you can use settings like the following that if the raspberry pi starts without any display headless, the gui can be used external like VNC does:
framebuffer_width=1280
framebuffer_height=1024
hdmi_force_hotplug=1
#dtoverlay=vc4-fkms-v3dAfter that you can choose if you want to use SSH or VNC.
Find more under
- ROS Melodic on Raspberry Pi 4 Debian Buster + RPLIDAR A1M8
- ROSberryPi: Installing ROS Melodic on the Raspberry Pi
- ROSberryPi: Installing ROS Kinetic on the Raspberry Pi
These instructions assume that Raspbian Buster is being used as the OS on the Raspberry Pi 4 Model B.
First install repository key:
$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
$ sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654Now, make sure your Debian package index is up-to-date:
$ sudo apt-get update
$ sudo apt-get upgrade$ sudo apt install -y python-rosdep python-rosinstall-generator python-wstool python-rosinstall build-essential cmake$ sudo rosdep init
$ rosdep updateNow, we will download and build ROS Melodic.
In order to build the core packages, you will need a catkin workspace. Create one now:
$ mkdir -p ~/ros_catkin_ws
$ cd ~/ros_catkin_wsDesktop Install: includes GUI tools, such as rqt, rviz, and robot-generic libraries. Might be better choice for beginners to ROS.
$ rosinstall_generator desktop --rosdistro melodic --deps --wet-only --tar > melodic-desktop-wet.rosinstall
$ wstool init -j8 src melodic-desktop-wet.rosinstallThis will add all of the catkin or wet packages in the given variant and then fetch the sources into the ~/ros_catkin_ws/src directory. The command will take a few minutes to download all of the core ROS packages into the src folder. The -j8 option downloads 8 packages in parallel.
Before you can build your catkin workspace, you need to make sure that you have all the required dependencies. We use the rosdep tool for this.
Resolving Dependencies with rosdep The dependencies should be resolved by running rosdep:
$ cd ~/ros_catkin_ws
$ rosdep install -y --from-paths src --ignore-src --rosdistro melodic -r --os=debian:busterThis will look at all of the packages in the src directory and find all of the dependencies they have. Then it will recursively install the dependencies.
The --from-paths option indicates we want to install the dependencies for an entire directory of packages, in this case src. The --ignore-src option indicates to rosdep that it shouldn't try to install any ROS packages in the src folder from the package manager, we don't need it to since we are building them ourselves. The --rosdistro option is required because we don't have a ROS environment setup yet, so we have to indicate to rosdep what version of ROS we are building for. Finally, the -y option indicates to rosdep that we don't want to be bothered by too many prompts from the package manager.
After a while rosdep will finish installing system dependencies and you can continue.
Once you have completed downloading the packages and have resolved the dependencies, you are ready to build the catkin packages.
Invoke catkin_make_isolated:
$ sudo ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --install-space /opt/ros/melodic -j8Now ROS should be installed! Remember to source the new installation. Source the setup.bash in the ~/.bashrc, so that ROS environment variables are automatically added to your bash session every time a new shell is launched:
$ echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrcFind more under http://wiki.ros.org/melodic/Installation/Source
cd ~/ros_catkin_ws/To update your workspace, first move your existing rosinstall file so that it doesn't get overwritten, and generate an updated version. For simplicity, we will cover the destop-full variant. For other variants, update the filenames and rosinstall_generator arguments appropriately.
$ mv -i melodic-desktop-full.rosinstall melodic-desktop-full.rosinstall.old
$ rosinstall_generator desktop_full --rosdistro melodic --deps --tar > melodic-desktop-full.rosinstallThen, compare the new rosinstall file to the old version to see which packages will be updated:
$ diff -u melodic-desktop-full.rosinstall melodic-desktop-full.rosinstall.oldIf you're satisfied with these changes, incorporate the new rosinstall file into the workspace and update your workspace:
$ vcs import src < melodic-desktop-full.rosinstallNow that the workspace is up to date with the latest sources, rebuild it:
sudo ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --install-space /opt/ros/melodic -j8Once your workspace has been rebuilt, you should source the setup files again:
$ source /opt/ros/melodic/setup.bashOr just reload the ~/.bashrc file:
$ source ~/.bashrcYou may add additional packages to the installed ros workspace that have been released into the ros ecosystem. First, a new rosinstall file must be created including the new packages (Note, this can also be done at the initial install). For example, if we have installed ros_comm, but want to add ros_control and joystick_drivers, the command would be:
$ cd ~/ros_catkin_ws
$ rosinstall_generator ros_comm ros_control joystick_drivers --rosdistro melodic --deps --wet-only --tar > melodic-custom_ros.rosinstallYou may keep listing as many ROS packages as you'd like separated by spaces.
Next, update the workspace with wstool:
$ wstool merge -t src melodic-custom_ros.rosinstall
$ wstool update -j8 -t srcAfter updating the workspace, you may want to run rosdep to install any new dependencies that are required:
$ rosdep install --from-paths src --ignore-src --rosdistro melodic -y -r --os=debian:busterFinally, now that the workspace is up to date and dependencies are satisfied, rebuild the workspace:
$ sudo ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --install-space /opt/ros/melodic -j8Once your workspace has been rebuilt, you should source the setup files again:
$ source /opt/ros/melodic/setup.bashOr just reload the ~/.bashrc file:
$ source ~/.bashrcDon`t follow the raspicam_node installation guide because the raspicam_node was built for the kinetic distribution. You have to use the new build raspica_node. For the melodic distribution you have to do the following workaround:
The original node is primarily supported on ROS Kinetic, and Ubuntu 16.04. This instructions are made for ROS Melodic and Raspbian Buster. So we'll have to build it from source.
Make sure that your user is in the video group by running groups|grep video. If not type following in your command line: sudo usermod -a -G video <username>
Now install following dependecies for Python:
sudo -H pip install -U pygithub
sudo -H pip install -U pygithub3
sudo -H pip install -U chainercv
You have first to install pygithub because pygithub 3 couldn't be found if not.
Go to your ros_catkin_ws cd ~/ros_catkin_ws/.
After that we have to use the rosinstall_generator to add some new ROS Packages.
rosinstall_generator compressed_image_transport --rosdistro melodic --deps --wet-only --tar > melodic-compressed_image_transport-wet.rosinstall
rosinstall_generator camera_info_manager --rosdistro melodic --deps --wet-only --tar > melodic-camera_info_manager-wet.rosinstall
rosinstall_generator dynamic_reconfigure --rosdistro melodic --deps --wet-only --tar > melodic-dynamic_reconfigure-wet.rosinstall
Then we have to merge it to our current ROS installation.
wstool merge -t src melodic-compressed_image_transport-wet.rosinstall
wstool merge -t src melodic-camera_info_manager-wet.rosinstall
wstool merge -t src melodic-dynamic_reconfigure-wet.rosinstall
And then we have to update it with wstool update -t src
Then you have to install the rosdeps again:
rosdep install --from-paths src --ignore-src --rosdistro melodic -y -r
Now we can run the build process again with:
sudo ./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release --install-space /opt/ros/melodic
Go to your catkin_ws with cd ~/catkin_ws/src
Download the source for this node by running
git clone https://github.com/Michdo93/raspicam_node.git
Normally you should skip the follwing part with updating the rosdep:
There are some dependencies that are not recognized by ros, so you need to create the file /etc/ros/rosdep/sources.list.d/30-ubiquity.list and add this to it.
yaml https://raw.githubusercontent.com/UbiquityRobotics/rosdep/master/raspberry-pi.yaml
Then run rosdep update.
If skipping this part does not work try to install it like in the description above or to install it with the apt-package manager like this:
sudo apt-get install libraspberrypi0
sudo apt-get install libraspberrypi-dev
sudo apt-get install libpigpio-dev
sudo apt-get install libpigpiod-if-devNormally on a Raspberry Pi with a Raspbian OS this packages should be installed by default.
Install the ros dependencies,
cd ~/catkin_ws
rosdep install --from-paths src --ignore-src --rosdistro=melodic -y
Compile the code with catkin_make.
Later we will do some special configuration stuff that we can use ROS under Python 3 and that the RobotCar can find its needed control computer respectively operating computer.
For the control computer or operating computer a virtual machine is recommended. Here you can install Ubuntu like Ubuntu Bionic Beaver. Then you can install ROS like in the following [instruction}(http://wiki.ros.org/melodic/Installation/Ubuntu). It's much easier.
Configure your Ubuntu repositories to allow "restricted," "universe," and "multiverse." You can follow the Ubuntu guide for instructions on doing this.
Setup your computer to accept software from packages.ros.org.
sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
If you experience issues connecting to the keyserver, you can try substituting hkp://pgp.mit.edu:80 or hkp://keyserver.ubuntu.com:80 in the previous command.
Alternatively, you can use curl instead of the apt-key command, which can be helpful if you are behind a proxy server:
curl -sSL 'http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC1CF6E31E6BADE8868B172B4F42ED6FBAB17C654' | sudo apt-key add -
First, make sure your Debian package index is up-to-date:
sudo apt update
There are many different libraries and tools in ROS. We provided four default configurations to get you started. You can also install ROS packages individually.
In case of problems with the next step, you can use following repositories instead of the ones mentioned above ros-shadow-fixed
Desktop-Full Install: (Recommended) : ROS, rqt, rviz, robot-generic libraries, 2D/3D simulators and 2D/3D perception
sudo apt install ros-melodic-desktop-full
It's convenient if the ROS environment variables are automatically added to your bash session every time a new shell is launched:
echo "source /opt/ros/melodic/setup.bash" >> ~/.bashrc
source ~/.bashrc
If you have more than one ROS distribution installed, ~/.bashrc must only source the setup.bash for the version you are currently using.
If you just want to change the environment of your current shell, instead of the above you can type:
source /opt/ros/melodic/setup.bash
If you use zsh instead of bash you need to run the following commands to set up your shell:
echo "source /opt/ros/melodic/setup.zsh" >> ~/.zshrc
source ~/.zshrc
Up to now you have installed what you need to run the core ROS packages. To create and manage your own ROS workspaces, there are various tools and requirements that are distributed separately. For example, rosinstall is a frequently used command-line tool that enables you to easily download many source trees for ROS packages with one command.
To install this tool and other dependencies for building ROS packages, run:
sudo apt install python-rosdep python-rosinstall python-rosinstall-generator python-wstool build-essential
Before you can use many ROS tools, you will need to initialize rosdep. rosdep enables you to easily install system dependencies for source you want to compile and is required to run some core components in ROS. If you have not yet installed rosdep, do so as follows.
sudo apt install python-rosdep
With the following, you can initialize rosdep.
sudo rosdep init
rosdep update
It is recommended to integrate the robot car as network drive at the operating PC. The Samba server is installed for this purpose. Some precautions must be taken for both the server and the client. Logically, the robot car provides the server.
On the Robotcar execute a terminal and install samba with:
sudo apt-get install samba samba-common-bin
After that you have to change die configuration file with sudo nano /etc/samba/smb.conf. For example, if you want to share the whole RobotCar, you must share /home/pi off the user pi. An example could look like this:
[global]
netbios name = Pi
server string = The RobotCar File System
workgroup = WORKGROUP
[HOMEPI]
path = /home/pi
comment = No comment
browsable = yes
writable = Yes
create mask = 0777
directory mask = 0777
public = no
Then you have to set a samba password with sudo smbpasswd -a <username>. So for the given example with the user pi sudo smbpasswd -a pi. You have to enter a password and press enter. As example the RobotCar uses robotcar as samba password.
Then you have to restart the samba service with sudo /etc/init.d/smbd restart.
On the operating computer or control computer you have to open a terminal and install following:
sudo apt-get install cifs-utils
For mouting the network drive you have to create a folder like with sudo mkdir /mnt/RobotCar. Then you can mount it with:
sudo mount -t cifs -o username=<username>,password=<samba password>,uid=1000 //<hostname>/<folder> /mnt/RobotCar/
As example with:
sudo mount -t cifs -o username=pi,password=robotcar,uid=1000 //robotcar/robotcar /mnt/RobotCar/
If you loose the network connection it could be that you have redone the last step. After that it is possible to use as example Visual Studio Code from your operating computer or control computer to change files on the RobotCar.
Normally Python should be installed on Raspbian. You can check it with python3 --version. If not you have to do following:
$ sudo apt-get update
$ sudo apt-get install python3.7
For Raspbian a Python library for using the GPIO Pins should be installed. If not, you can install it with:
sudo apt-get install python-rpi.gpio
If you want to use pip you have to install it with:
sudo apt-get install python-pip
sudo apt-get install python3-pip
As example you can install via pip3 following:
pip3 install readchar
pip3 install flask
The readchar library could be used to read the input from the keyboard. The flask web framework could be used that the RobotCar as example could be used as a webserver. This could be helpful to control the RobotCar if you want to create a corresponding program.
If you want to create a test picture you could use:
raspistill -o testbild.jpg
The picture testbild.jpg should be created in your current folder.
If you want to create a simple video of 20 seconds in the h264 codec you could run:
raspivid -o testvideo.h264 -t 20000
The picture testvideo.h264 should be created in your current folder.
For using the raspicam without ROS you can install Cheese with:
sudo apt-get install cheese
Maybe a helpful installation could be he mjpg-streamer. It allows us to show a movie stream from the camera via a web based url. So you can see what the camera sees. This could be helpful to conrol the Robotcar.
At first you have to install following packages:
sudo apt-get install libjpeg8-dev
sudo apt-get install cmake
The you have to go to cd /opt/ and download it with:
sudo wget https://custom-build-robots.com/mjpg-streamer.zip
After that you have to unzip the archive and open the unzipped folder and build the mjpg-streamer:
cd /opt/
sudo unzip mjpg-streamer.zip
cd mjpg-streamer
sudo make
sudo make install
Then you have to load the kernel module for Video4Linux driver version 2 (v4l2) with sudo modprobe bcm2835-v4l2.
You can check it with sudo lsmod or with sudo nano /etc/modules:
i2c-dev
bcm2835-v4l2
The we have to finde the Device name of the camera with ls /dev/vid*. This could be as example /dev/video0.
To configure the mjpg-streamer change it with nano /opt/mjpg-streamer/start.sh and extend the following line
./mjpg_streamer -i "./input_uvc.so" -o "./output_http.so -w ./www"
with
./mjpg_streamer -i "./input_uvc.so" -d /dev/video0 -r 800x640 -f 25 -o "./output_http.so -w ./www"
So you added the device video0 with resolution 800x640 and framerate 25.
The go to cd /opt/mjpg-streamer/ enter sudo modprobe bcm2835-v4l2 and restart it with sudo sh ./start.sh.
Now you can see it under http://<ip address of the robotcar>:8080
That it runs on every startup you have to copy it to /etc/init.d with sudo cp /opt/mjpg-streamer.sh /etc/init.d/mjpg-streamer.sh.
Then change the execution permissions with sudo chmod +x /etc/init.d/mjpg-streamer.sh.
Enter in rc.d for auto start
sudo update-rc.d mjpg-streamer.sh defaults
Sign out again (dont do it, if you want it to run after the startup.)
sudo update-rc.d mjpg-streamer.sh remove
Check if started
sudo reboot
systemctl list-units -t service
Start manually
sudo service mjpg-streamer.sh start
Stop
sudo service mjpg-streamer.sh stop
Restart
sudo service mjpg-streamer.sh restart
The time can be synchronized via an NTP server using the NTP client. This can be installed as follows:
sudo apt-get install ntp
sudo dpkg-reconfigure tzdata
Then you set the-timezone as example to Europe/Berlin with:
sudo timedatectl set-timezone Europe/Berlin
sudo /etc/init.d/ntp restart
You can check the time synchronization with:
date
Maybe there are multiple NTP servers available. To check if it only uses one server you can run:
ntpg -pn
If there are more than one NTP server you can change it with nano /etc/ntp.conf:
# SHM 0
server 127.127.28.0
fudge 127.127.28.0 refid GPSa
# SHM 1
server 127.127.28.1
fudge 127.127.28.1 refid GPSp
In the given example you can delete one entry. After that you have to restart it again and check again if there is only one server:
sudo /etc/init.d/ntp restart
ntpq -pn
We need the I2C bus of the raspberry pi. After enabling it we have to instal it with:
sudo apt-get install i2c-tools
sudo apt-get install python-smbus
To check if it is working and if connected devices could be found you can run:
i2cdetect -y 1
The necessary GPS programs such as the Python libraries, the gpsd daemon server and the xGPS program are all installed on your Raspberry Pi together with the following command:
sudo apt-get install gpsd gpsd-clients python-gps
sudo apt-get install python-gi-cairo
Further informations you can found here.
Enable it: sudo systemctl enable gpsd.socket
Start it: sudo systemctl start gpsd.socket
Restart it: sudo systemctl restart gpsd.socket
Check status: sudo systemctl status gpsd.socket
Modify the "DEVICE" parameter according to the name of serial port in /dev folder. It is usually named "/dev/ttyUSB0" if you connect it to Raspberry Pi via USB cable.
sudo nano /etc/default/gpsd
As example you can see something like this:
# Default settings for the gpsd init script and the hotplug wrapper.
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
DEVICES="/dev/ttyUSB0"
# Other options you want to pass to gpsd
GPSD_OPTIONS="-F /var/run/gpsd.sock"
The restart the service with sudo systemctl restart gpsd.socket
Finally, use this command to get information from GPS module:
sudo cgps -s
Start the gpsd server daemon with the following command in the terminal window:
sudo gpsd -b /dev/ttyACM0 -F /var/run/gpsd.sock -G
You could test the gps with the gps-test.py file inside robotcar/test:
sudo python3 gps-test.py
In order to be able to calibrate the magnetometer of the Raspberry Pi Sense HAT later, you must install the Octave programme. Octave is an interactive scripting language which can be used to solve problems from numerical mathematics together with the Raspberry Pi Sense HAT, e.g. calculating the local magnetic field of the earth.
sudo apt-get install octave
Further informations to the Sense HAT you can found in the API. You have to install and restart the Sense HAT with following commands:
sudo apt-get install sense-hat
sudo reboot
The magnetometer must be calibrated for the respective place of use, since the earth's magnetic field is location-dependent and there may also be interference from the environment. You can do it with:
cd /usr/share/librtimulib-utils/RTEllipsoidFit
sudo RTIMULibCal
Then you have to press m for calibrate Magnetometer.
Then roll, turn and tilt the RobotCar as described. Repeat the various movements alternately until the numbers in the terminal window no longer change. The sequence of the turn, tilt and roll movements is irrelevant. If the numbers remain the same, calibration is complete and by entering the letter S you save the recorded data in the configuration file RTIMULib.ini. Then press X to exit the RTIMULibCal program.
To do this, press key A, follow the description displayed and calibrate the accelerometer in the same steps as you did for the magnetometer.
Creates /usr/share/librtimulib-utils/RTEllipsoidFit/RTIMULib.ini. Copy it with cp /usr/share/librtimulib-utils/RTEllipsoidFit/ /etc/RTIMULib.ini.
The RTIMULib.ini file is expected by the Sense-HAT-API in the /etc/ folder. If you use the Sense-HAT-API in a Python program that also uses the magnetometer and accelerometer, the RTIMULib.ini file is read from there.
The Raspberry-Pi-Sense-HAT-API creates a copy of the file RTIMULib.ini in the folder /home/pi/.config/sense_hat. Always delete the RTIMULib.ini file there when you have performed a new calibration.
The program display updates every 0.1 seconds and shows you how much the orientation of the robot car deviates from the north. By turning the robot car, you can orient it towards the south.
Code OSS is nearly the same as Visual Studio Code. It could be installed as alternative text editor on the Raspberry Pi like following:
wget https://packagecloud.io/headmelted/codebuilds/gpgkey -O - | sudo apt-key add
curl -L https://code.headmelted.com/installers/apt.sh | sudo bash
If this does not work try it with:
wget -o - https://packagecloud.io/headmelted/codebuilds/gpgkey| sudo apt-key add -
sudo apt-get install code-oss=1.29.0-1539702286
You can open and edit a Python program with code-oss <filename.py> like code-oss main.py.
In order to share ROS, OpenCV and TensorFlow, a virtual environment must be created using Python 3. Unfortunately, the Edge TPU only works with Python 3, which means TensorFlow. ROS, on the other hand, only works under Python 2, but with a workaround it also works under Python 3. OpenCV can be run under Python 2 and Python 3 just like TensorFlow. However, the Edge TPU is required as a co-processor for TensorFlow, otherwise the performance would be insufficient.
First of all we install a few packages to resolve dependencies:
sudo apt-get install libhdf5-dev -y && sudo apt-get install libhdf5-serial-dev -y && sudo apt-get install tflibatlas-base-dev -y && sudo apt-get install libjasper-dev -y && sudo apt-get install libqtgui4 -y && sudo apt-get install libqt4-test -y
OpenCV cannot be used in one of the 4 versions due to ROS. Therefore we install it in the following version:
pip3 install opencv-python==3.4.3.18
After that we install with pip3 install matplotlib matplotlib.
We can check the installation with:
python3 -c "import cv2"
python3 -c "import numpy"
python3 -c "import matplotlib"
Then we install the Edge TPU driver:
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add –
sudo apt-get update
sudo apt-get install libedgetpu1-std
sudo apt-get install libedgetpu1-max
sudo apt-get install python3-edgetpu
This would allow OpenCV and TensorFlow to be used outside a virtual environment. So you can run or test programs separately.
In the next step we install virtualenv and the virtualenvwrapper:
sudo pip install virtualenv virtualenvwrapper==4.8.4
In the .bashrc file we add something like this:
# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.7
source /usr/local/bin/virtualenvwrapper.sh
workon robotcar
Then we create our virtualenv called robotcar:
mkvirtualenv robotcar -p python3
You can check the installation with:
cd ~/.virtualenvs/coral/lib/python3.7/site-packages
ln -s /usr/lib/python3/dist-packages/edgetpu/ edgetpu
Then you have to activate the virtualenv with:
workon robotcar
Now you can use pip instead of pip3 or python instead of python3 because the virtualenv runs only under Python 3.
Afterwards some more packages have to be installed...
pip install "picamera[array]"
pip install opencv-python==3.4.3.18
pip install matplotlib
pip install imutils
pip install tensorflow==2.0.0
pip install keras –no-use-pep517
pip install pillow
pip install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_armv7l.whl
Then we have to create a folder where we want to install TensorFlow Lite (tflite). It's the mobile Version of TensorFlow whiche are used on the TPU.
mkdir coral && cd coral
git clone https://github.com/google-coral/tflite.git
cd tflite/python/examples/classification
bash install_requirements.sh
You can test this installation with:
python3 classify_image.py \
--model models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
--labels models/inat_bird_labels.txt \
--input images/parrot.jpg
=====================
The simulation platform uses four differents software modules which also needs further catkin packages.
| Software Module | System | Description |
|---|---|---|
| robotcar | RobotCar | The robotcar module provides all drivers for actors and sensors. Also it provides all needed Libraries and configurations. Filters like the kalman filter are also provided. Programs for configuring and resetting the sensors are available as well as for testing the motor and controller. Service files for systemd and start-up are also available. This software module implements the functionality of the robot car. |
| robotcar-pkg | RobotCar | Via the ROS Publisher-Subscriber pattern, the robotcar-pkg enables the exchange of information. Publishers provide the information to ADAS and via subscriber an ADAS can intervene in the driving of the robot car. This software module therefore enables the necessary communication for the functionality. |
| robotcar_controller | control computer respectively operating computer | The controller forwards appropriate control commands to the robot car, which should make it possible to move the robot car purposefully. For this purpose, primarily the motor and the steering are controlled. An extension, for example to move the camera or switch ADAS on and off, would be conceivable in the future. |
| robotcar_msgs | Robotcar and control computer respectively operating computer | The robotcar_msgs extend the ROS message types with custom message types for complete communication. For new ADAS these may have to be extended. On every computer that wants to participate in the communication, these must be installed additionally. |
Further packages:
- With the beginner_tutorials you can learn the basic concepts of the RobotCar.
- With the robotcar_subscriber you can subscribe different informations from the RobotCar. It could be used as blue print for ADAS.
- With the robotcar_sensorfusion_examples you can learn how to use as example a simple kalman filter for sensor fusion. It could be used as blue print for ADAS.
- The std_header_msgs could be used as example for sensor fusion because the sensor fusion needs timestamps which are missing in the std_msgs from ROS.
- The raspicam_node is needed to use the raspicam with ROS. So the robot car definitely needs this package.
- The raspicam_node_saver is needed to save frames from the raspicam_node as individual JPEG frames or as avi video. It can therefore be used, for example, to record training data for machine learning.
- The robotcar_plotter shows various curves for sensor information or diagrams for the position and orientation of the RobotCar, e.g. compass or 3D gyroscope, etc.
After the installation of the necessary software and frameworks, basic precautions have to be taken for the simulation platform so that the operating computer and RobotCar can find each other in the network, ROS can be used accordingly in the command line and the virtual environment OpenCV and TensorFlow, as well as other packages can be integrated. For this purpose, the virtual environment is started and each terminal window executes it automatically.
So we edit our terminal configuration with sudo ~/.bashrc:
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
workon coral
source /opt/ros/melodic/setup.bash
source /home/ros/catkin_ws/devel/setup.bash
. ~/catkin_ws/devel/setup.bash
export ROS_HOSTNAME="$(hostname -f)"
export ROS_MASTER_URI=http://"$(hostname -f)":11311
export ROS_IP="$(hostname -I | awk '{print $1;}')"
export ROS_PYTHON_VERSION=3
As example it works on a virtualenv named coral.
The same applies to the robot car as to the operating computer. There are a few small changes.
So we edit our terminal configuration with sudo ~/.bashrc:
# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh
#workon robotcar
source $WORKON_HOME/robotcar/bin/activate
#export PATH=/usr/bin/python2.7:$PATH
source /opt/ros/melodic/setup.bash
source /home/pi/catkin_ws/devel/setup.bash
export ROS_HOSTNAME="$(hostname -f)"
export MASTER_URI_NAME=ros-melodic-master
export ROS_MASTER_URI=http://$MASTER_URI_NAME:11311
#export ROS_MASTER_URI=http://141.28.75.144:11311
export ROS_IP="$(hostname -I | awk '{print $1;}')"
You work on a virtualenv called robotcar.
If multiple RobotCars are available you have to change the hostname.
You have to go into the src folder of your catkin workspace with cd ~/catkin_ws/src and clone following repositories:
git clone https://github.com/Michdo93/robotcar_controller.git
git clone https://github.com/Michdo93/robotcar_msgs.git
git clone https://github.com/Michdo93/beginner_tutorials.git
git clone https://github.com/Michdo93/robotcar_subscriber.git
git clone https://github.com/Michdo93/robotcar_sensorfusion_examples.git
At first we want to clone this repository. For this we go to the root directory with cd ~ and clone it with:
https://github.com/Michdo93/robotcar.git
Then we have to install the necessary libraries. So we go /robotcar/lib with cd robotcar/lib. You have to make sure that you are inside of the virtualenv. If not you have to enter workon robotcar.
At first we began with the MCP3008 library:
cd Adafruit_Python_MCP3008
sudo python setup.py install
After that we have to install the Adafruit_Python_SSD1306 library:
cd ../Adafruit_Python_SSD1306
sudo python setup.py install
Then we install the piVirtualWire library:
cd ../piVirtualWire
sudo apt-get install pigpio
sudo systemctl start pigpiod
sudo systemctl enable pigpiod
The we want to install the vl53l1x-python library for our Time-of-Flight sensors:
cd ../vl53l1x-python
sudo python setup.py install
At least we install an older version of the Adafruit_Python_PCA9685 library:
cd ../Adafruit_Python_PCA9685-master
sudo python setup.py install
After that we create the startup of the robotcar. We could go to /robotcar/startup. You have to execute:
sudo cp ~/robotcar/startup/robotcar.service /etc/systemd/system/robotcar.service
sudo cp ~/robotcar/startup/clear.service /etc/systemd/system/clear.service
cd /etc/systemd/system/
sudo systemctl enable robotcar.service
sudo systemctl enable clear.service
You can control the service files with:
sudo systemctl start robotcar.service
sudo systemctl stop robotcar.service
sudo systemctl enable robotcar.service
sudo systemctl disable robotcar.service
sudo systemctl start clear.service
sudo systemctl stop clear.service
sudo systemctl enable clear.service
sudo systemctl disable clear.service
The service files switch off the LEDs of the Sense HAT, resolve the I2C conflict of both ToF sensors and show on the OLED display the host name, IP address and the user name of the robot car where the files clear.py, vl53l1x.py and robot.py are executed at system startup.
Next, we go into the src directory of the Catkin workspace with cd ~/catkin_ws/src and clone all necessary repositories:
git clone https://github.com/Michdo93/robotcar-pkg.git
git clone https://github.com/Michdo93/robotcar_msgs.git
git clone https://github.com/Michdo93/beginner_tutorials.git
git clone https://github.com/Michdo93/robotcar_subscriber.git
git clone https://github.com/Michdo93/robotcar_sensorfusion_examples.git
git clone https://github.com/Michdo93/std_header_msgs.git
The raspicam_node should be installed during the ROS installation. If not you can clone it with:
https://github.com/Michdo93/raspicam_node.git
Now everything should be installed.
As last step you can configurate the servos of the steering and pan-tilt-bracket by running (inside the robotcar folder):
python configurate_servos.py