Automated Camera Stabilization and Calibration for Intelligent Transportation Systems - Static Calibration

• See the project website of Providentia++
• See the overview repository for general information about my Guided Research
• See the website for information and evaluation results of the project

---

Introduction

We track and predict the real-world location of vehicles in the test area to pass this information to the drivers and autonomous vehicles. To accurately predict the locations the system needs to be calibrated precisely towards a global reference frame. This is a time consuming process that often has to be done by hand. The cameras translational, rotational and intrinsic parameters decalibrate over time due to environmental influences on the mounting constructions and gantry bridges as well as from the natural wear of the materials.

Within the project high definition maps (HD maps) of the enclosed environment are used extensively. These HD maps offer approximations of the real-world positions of the highway lanes, the gantry bridges, objects like poles and permanent delineators and traffic signals like speed limits or exit markers. We use this spatial information and a mapping from the objects to pixels in the video frame to solve a Bundle Adjustment (BA) problem by minimizing the reprojection-error. We jointly optimize for the cameras intrinsic and extrinsic parameters as well as the real-world locations to recover the camera poses from the observations.

Please see the report for in depth information about the pose estimation procedure

---

Before calibration

Points of permanent delineators from the HD map mapped to pixel locations (green) and points without known corresponding pixels (red) rendered by a poorly calibrated camera model. The mapping from the projected points to their expected pixels is drawn in light blue.

After calibration

The same points after the calibration procedure. The rendered positions of the mapped points align with their respective pixels. The drawn mapping disappears as the distances approach 0.

---

Running in Standalone Mode

See below for the necessary dependencies.

Compiling

mkdir build && cd build
cmake ..
cmake --build . -j8

CMake Configure Options

# Default is OFF for all options

-DWITH_TESTS=ON/OFF     # Build with tests. 

-DWITH_OPENCV=ON/OFF    # Build with OpenCV. 
                        # Set to ON to render the projected world objects during optimization. 

Configuration

You have to create a <config>.yaml file to provide the necessary parameters to the static calibration program.

See config/ for examples. config/s50_s_near/config.yaml is fully commented.

Running

./app/StaticCalibration --config <path_to_config_file>

Testing

ctest --verbose

---

Input Data

The calibration requires a 2D-3D mapping from pixels to objects to estimate the 6DoF pose of the camera.

Examples to the files can be found in config/ and misc/.

Objects

objects:
  - # Must match with the ids in the pixel file. This establishes the mapping.
    id: 4008327
    # Has to be pole. No other type is currently implemented.
    type: pole
    # Has to be permanentDelineator. No other name is currently implemented.
    name: permanentDelineator
    # [XYZ | East/North/Height] coordinates in a right-handed coordinate system. 
    utm_coord: [ -878.08065099595115, 858.9148813476786, -1.5917528217373729 ]
    # The height in the same coordinate system as in utm_coord.
    height: 1.135
  - …

OpenDRIVE Parser

To facilitate the creation of the objects_file, we have released an OpenDRIVE parser that converts from the OpenDRIVE V1.4 standard to our internally used format.
See the README in the project for the usage.

---

Pixels

-p [ --pixels_file ] <filename>

The pixels_file is required to be in the YAML format as specified:

- # Must match with the ids in the objects_file. This establishes the mapping.
  id: 4008327
  # The UV pixel coordinates of the object.
  pixels:
    - [ 13, 373 ]
    - [ 13, 374 ]
    - [ 14, 374 ]
    - [ 13, 375 ]
    - [ 14, 375 ]
    - …
- …

Frame Annotation Tool

To facilitate the creation of the pixels_file, we have released an Annotation Tool that can be used to mark pixels in a frame and outputs them in the required format.
See the README in the project for the usage.

---

Dependencies

Dependency	Usage	Installation	Required
Boost	Command line parsing	sudo apt install libboost-all-dev	Yes
Ceres Solver	Non-linear optimization	extern/setup_ceres.sh	Yes
OpenCV	Rendering during evaluation	extern/setup_opencv.sh Please only run if you don't use OpenCV elsewhere. This might cause problems if you have your own OpenCV setup installed, as it compiles OpenCV with the minimal flags needed and overrides the system installation.	-DWITH_OPENCV=ON

Internal Dependencies

These dependencies are pulled by CMake when the project is built. You do not have to install them manually.

Dependency	Usage	Required
YAML-CPP	YAML parser for objects and pixels	Yes
TinyXML2	XML parser & writer for conversion to ROS node	Yes
GoogleTest	Google unit testing framework	-DWITH_TESTS=ON