[IEEE TITS 2023] DeepSORVF: Deep Learning-based Simple Online and Real-Time Vessel Data Fusion Method

---

Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways
Yu Guo, Ryan Wen Liu*, Jingxiang Qu, Yuxu Lu, Fenghua Zhu*, Yisheng Lv (* indicates corresponding author)
IEEE Transactions on Intelligent Transportation Systems

Introduction: In this work, we first extract the AIS- and video-based vessel trajectories, and then propose a deep learning-enabled asynchronous trajectory matching method (named DeepSORVF) to fuse the AIS-based vessel information with the corresponding visual targets. In addition, by combining the AIS- and video-based movement features, we also present a prior knowledge-driven anti-occlusion method to yield accurate and robust vessel tracking results under occlusion conditions.

---

Flowchart

Figure 1. The architecture of the proposed deep learning-based simple online and real-time vessel data fusion method.

Figure 2. The flowchart of anti-occlusion tracking method for video-based vessel trajectory extraction.

Environment

• Python3.7
• easydict 1.11
• geopy 2.4.1
• pyproj 3.2.1
• fastdtw 0.3.4
• pytorch 1.13.1
• cuda 11.7
• pandas 1.3.5
• numpy 1.21.6

Running

• Save ckpt.t7 to DeepSORVF/deep_sort/deep_sort/deep/checkpoint/ folder.
• Save YOLOX-final.pth to DeepSORVF/detection_yolox/model_data/ folder.
• Set data dir by parser.add_argument("--data_path", type=str, default = './clip-01/', help='data path').
• Run main.py.

draw_org.py is used to simultaneously visualize the ais-based trajectory (blue line), target detection box (red box), and fusion results (black text). It can be enabled by modifying import draw in main.py to import draw_org.

Test Data: clip-01

FVessel: Benchmark Dataset for Vessel Detection, Tracking, and Data Fusion

The FVessel benchmark dataset is used to evaluate the reliability of AIS and video data fusion algorithms, which mainly contains 26 videos and the corresponding AIS data captured by the HIKVISION DS-2DC4423IW-D dome camera and Saiyang AIS9000-08 Class-B AIS receiver on the Wuhan Segment of the Yangtze River. To protect privacy, the MMSI for each vessel has been replaced with a random number in our dataset. As shown in Figure 1, these videos were captured under many locations (e.g., bridge region and riverside) and various weather conditions (e.g., sunny, cloudy, and low-light).

Figure 3. Some samples of the FVessel dataset, which contains massive images and videos captured on the bridge region and riverside under sunny, cloudy, and low-light conditions.

Performance on FVessel_V1.0

Name	MOFA (%)	IDP (%)	IDR (%)	IDF (%)	Video
video-01	79.94	89.35	90.76	90.05
video-02	73.19	83.27	91.60	87.23
video-03	96.45	99.23	97.20	98.20
video-04	98.08	99.45	98.63	99.03
video-05	89.19	93.46	95.91	94.67
video-06	91.17	96.04	95.08	95.56
video-07	96.81	99.59	97.21	98.39
video-08	82.28	99.64	82.58	90.31
video-09	98.45	100.00	98.45	99.22
video-10	88.74	90.42	99.26	94.63
video-11	97.66	99.29	98.36	98.83
video-12	95.45	99.06	96.36	97.69
video-13	84.82	94.82	89.72	92.20
video-14	93.10	97.82	95.22	96.50
video-15	95.88	97.19	98.74	97.96
video-16	98.68	100.00	98.68	99.33
video-17	90.02	93.80	96.39	95.08
video-18	74.49	83.57	92.72	87.91
video-19	96.62	98.31	98.31	98.31
video-20	96.74	98.66	98.07	98.36
video-21	76.43	87.03	89.82	88.40
video-22	96.82	99.35	97.45	98.39
video-23	94.71	98.91	95.77	97.31
video-24	94.70	98.34	96.33	97.32
video-25	91.49	97.66	93.73	95.66
video-26	97.44	99.11	98.32	98.72
Average	91.13	95.90	95.41	95.59	...

Acknowledgements

We deeply thank Jianlong Su from the School of Computer and Artificial Intelligence in Wuhan University of Technology who performs the data acquisition and algorithm implementation works.

Citation

@article{guo2023asynchronous,
  title={Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways},
  author={Guo, Yu and Liu, Ryan Wen and Qu, Jingxiang and Lu, Yuxu and Zhu, Fenghua, and Lv, Yisheng},
  journal={IEEE Transactions on Intelligent Transportation Systems},
  year={2023}
}

Reference

https://github.com/bubbliiiing/yolox-pytorch

https://github.com/dyh/unbox_yolov5_deepsort_counting/tree/main/deep_sort

Visitor count