Reconstruction and Synthesis of Lidar Point Clouds of Spray

This is the official code release of the paper Reconstruction and Synthesis of Lidar Point Clouds of Spray by Yi-Chien Shih, Wei-Hsiang Liao, Wen-Chieh Lin, Sai-Keung Wong and Chieh-Chih Wang, acceptd at IEEE Robotics and Automation Letters (RA-L) 2022.

Acknowledgment

This project refers to the manipulation of point cloud defined in PointCloud by MaikelH, which is licensed under LGPL-2.1. We didn't modify the content but included the data structure in this project.

We would like to thank the authors of rapidjson(MIT) by Tencent, KDTree(MIT) by Vili Volčini, VehicleTools(MIT) by Unity Technologies, and datasacks(BSD) by Kurt Dekker for their great work.

Citation

If you find this work useful, please consider citing our paper.

@ARTICLE{9705289,
  author={Shih, Yi-Chien and Liao, Wei-Hsiang and Lin, Wen-Chieh and Wong, Sai-Keung and Wang, Chieh-Chih},
  journal={IEEE Robotics and Automation Letters}, 
  title={Reconstruction and Synthesis of Lidar Point Clouds of Spray}, 
  year={2022},
  volume={7},
  number={2},
  pages={3765-3772},
  doi={10.1109/LRA.2022.3148443}}

Overview

Environment

The code has been successfully tested on

• CPU: Intel Core i7-8700
• GPU: NVIDIA GeForce 1080
• RAM: 32GB
• OS: Ubuntu 18.04
• Unity Version: 2021.1.7f1
• Data sequences come from Waymo

Remember to install submodules listed in .gitmodules.

File Organization

There are 3 folders in the zipped file.

1. SpraySimulation
2. Tools

Table 1 shows the corresponding implementation for the different steps. Note that the name of the scene used under Unity simulation is SpraySimSimple.

Table 1. The phase and step are referred from Fig 2 in our method.

Phase	Step	Corresponding Algorithm Location
Simulation	Particle Trajectory Simulation	SpraySimulation/Assets/Script/SprayModel.cs
	Local Selection	SpraySimulation/Assets/Script/SprayModel.cs + the creation of filter_nodup.txt (mentioned below)
	Lidar Perception	SpraySimulation/Assets/Script/Lidar.cs
Data Processing	Synthesis	Tools/Synthesis_texture_based Tools/Synthesis_particle_based
	Global Refinement	Tools/Refinement

Getting Started

Simulation

Setup

Before running the simulation, we have to set some parameters. Below is a template of .env. Table 2 lists the meaning of each global parameter. Note that our configuration are palced at SpraySimulation/Config.

OutputPCDPath=
OutputFilterPath=
SprayDetectionJSONPath=
SimAugmentConfig=../Config/SimAugmentConfig.txt
NotSegmentedDataPath=
ReconstructRefPath=
ManipulationCheckConfig=../Config/ManipulationCheck.txt
DefaultFilter=

Create Plugins folder under Assets folder. Then create symbolic links for the 4 submodules installed eariler to (or copy to) the Assets/Plugins folder (i.e. SpraySimulation/Assets). The linking commands show in table 3.

Next, use Unity open this folder(i.e. SpraySimulation/).

If you open the project with safe mode, something is wrong. Please deal with them at first. Now it is time to start simulation. Table 4 shows the parameters we use under different simulation situations.

Table 2.

Global parameters: you should create a .env under SpraySimulation/Assets/Scripts/
OutputPCDPath	The folder stores point cloud output.
OutputFilterPath	The folder stores filter output.
SprayDetectionJSONPath	The folder stores spray detection results (in JSON).
SimAugmentConfig	The place stores configuration.
NotSegmentedDataPath	The folder stores the original un-segmented point cloud file (with .pcd extension).
ReconstructRefPath	The folder stores the original segmented point cloud file (with .pcd extension).
ManipulationCheckConfig	The place stored configuration.
DefaultFilter	The default folder stores default filter_nodup.txt/filter.txt.

Table 3. Symbolic links for the submodules.

L

First, cd to the directory SpraySimulation/Assets/Plugins/
Submodule	Link Command
PointCloud	ln -s ../../Submodules/PointCloud/PointCloud PointCloud
KDTree	ln -s ../../Submodules/KDTree/Assets/Datastructures KDTree
UnityVehicleTool	ln -s ../../Submodules/UnityVehicleTools UnityVehicleTool
datasacks	ln -s ../../Submodules/datasacks/datasacks/Assets/Datasack/Core/DatasackFormatting.cs .

Table 4. The game object is listed in Hierarchy window and attribute shows in Inspector window.

Game Object	Attribute	Situation
		Simulate with Random Particles	Simulate with Filtered Particles	Manipulation Check (Filter Particles)	Data Augmentation
Vehicles/EgoVehicleSpraySet	ParticleCountMultiplier	10,000	50,000	10,000	50,000
	VisibleParticleConfig	CloseToRealData	All	All	All
	SprayInitialStatusConfig	Random	Filtered	Random	Filtered
Lidar	Is Exporting Filter	true	false	false	false
Individual Setting		Set ReconstructRefPath(usually the ground-truth)and remained routing at Lidar.cs, line 169 (if needed).		Set Filter File Path at Vehicles/EgoVehicleSpraySet.	Set Filter File Path at Vehicles/EgoVehicleSpraySetcode>. Do the same setting under Assets/Prefab/CarSpraySet.
Common Setting		Set velocity of the ego vehicle in different sequences. See Table 5. Set wind field data according to the vehicle velocity at _WindFieldLoader. For example, the wind field of each sequence mentioned in our method shows in Table 4. Note that wind field data is placed at Assets/WindFieldData. See Table 4.

Table 5

Sequence Index / Name	Abbreviation	Velocity (m/s)	Correspond Wind Field
1 / segment-2974991090366925955	seg29	29	veh29_Scaled_with_Primitives0.19.csv
2 / segment-3132521568089292927	seg31	20	veh20_0.20.csv
3 / segment-5121298817582693383	seg51	29	veh29_Scaled_with_Primitives0.19.csv
4 / segment-11139647661584646830	seg111	21	veh21_0.20.csv
5 / segment-13830510593707564159	seg138	25	veh25_0.20.csv

Usage

1. Open this folder with Unity.
2. Open the scene named "SpraySimple".
3. Check the parameters mentioned above.
4. Navigation bar > Window > General > Recorder > Recorder Window, open the recorder window.
5. Set parameters in the recorder window. Table 6 shows parameters used in our method.
6. Start recording.

Table 6. Parameters for the recorder.

L

Parameter	Value
Recording Mode	Frame Interval
Start	0
End	220
Target FPS	Custom
Value	10

Data Processing - Synthesis

• This part should be build executed through the terminal.
• This part of details could apply to 2 systhesis scripts(i.e. Tools/Synthesis_texture_based and Tools/Synthesis_particle_based)
• Before synthesizing, generate filter_nodup.txt first. The followings show the steps:
  1. Open Tools/concat_file.py a. Set input_directory as your OutputFilterPath in simulation’s .env above. b. Set output_file name. Assume the name is “filter_new.txt” c. Execute this script.
  2. In command prompt, execute cat -n <your path to filter_new.txt> | sort -uk2 | sort -nk1 | cut -f2- > filter_nodup.txt. Done.
• Set the following parameters in main.cpp before building
  • filterFileName: the path to the filter_nodup.txt
• Other details (including in/output) are noted in main.cpp.
• An example of filter_nodup.txt (i.e. output) is provided.

Data Processing - Global Refinement

• This part should be build executed through the terminal.
• We rename the original .pcd files to <segment name>_<4 digit id>. For example, rename 1518656408.389677047.pcd to segment-2974991090366925955_0000.pcd or seg29_0000.pcd.
• We use CloudCompare in this part. Please make sure you’ve downloaded it.
• Set the following parameters in main.cpp before building
  • groundTruthDirectory: the folder stores renamed ground-truth .pcd
  • simulationResultDirectory: the folder stores simulated .pcd
• Other details (including in/output) are noted in main.cpp.
• An example of filter.txt (i.e. output) is provided.

Data Augumentation

• This part should be build executed through the terminal.
• Main file location: Tools/PCD_MergePointCloud
• Usage: Merge point clouds.
• Other details are noted in main.cpp.
• We use rapidjson as the JSON parser.

Contributor

• Yi Chien "Alan" Shih