Validating Synthetic Data for Perception in Autonomous Airport Navigation Tasks

Disclaimer: This project and all associated materials are currently under review for publication.

Authors: Miguel Ángel de Frutos Carro, Carlos Cerdán Villalonga, and Antonio Barrientos.

Affiliation: Centro de Automática y Robótica (UPM-CSIC), Universidad Politécnica de Madrid - Consejo Superior de Investigaciones Científicas, 28006 Madrid, Spain.

---

Abstract

Autonomous navigation within airport environments poses significant challenges, primarily due to the scarcity of accessible and labeled data for training autonomous systems. This study introduces an innovative approach to evaluate the performance of vision-based models trained on synthetic datasets. We conducted a comparative analysis of two datasets: a synthetic dataset that mirrors real airport scenarios, generated using Microsoft Flight Simulator 2020®, and a real-world dataset. The goal is to assess the effectiveness of simulated data in enhancing the training and validation processes for complex airport environments. Our methodology includes a comparative analysis using image processing techniques and object detection algorithms. The findings suggest that models trained on a mix of real and synthetic images perform better, showing improved adaptability and accuracy over those trained on a single dataset type. This research contributes significantly to the field of autonomous airport navigation, offering a cost-effective and practical solution to overcome dataset acquisition and algorithm validation challenges, thereby laying the groundwork for future advancements.

Keywords: Autonomous vehicles, airport markings recognition, computer vision, object detection, hybrid dataset, game engine.

---

Figures

Fig1. MSFS_AA (dataset_real) - Real Test Scenario

Fig2. REAL_AA (dataset_msfs) - MSFS Test Scenario

Fig3. Classes to be detected: (a) “Taxiway Lane”, (b) “Vertical Sign”, (c) “Person”, (d) Airplane, (e) “Horizontal Sign”, (f) “Runway Limit”, and (g) “Ground Vehicle”. Source: Author composition.

---

License:

GPU v3.0: https://github.com/astromaf/Synth_Airport_Taxii/blob/main/LICENSE

---

References:

1. Barresi, F. F., & Allasia, W. (2013). Airport markings recognition for automatic taxiing. In 2013 Conference on Design and Architectures for Signal and Image Processing (pp. 197–200).
2. Bauer, A., Trapp, S., Stenger, M., Leppich, R., Kounev, S., Leznik, M., Chard, K., & Foster, I. (2024). Comprehensive Exploration of Synthetic Data Generation: A Survey. http://arxiv.org/abs/2401.02524
3. Bisong, E. (2019). Google Colaboratory. In Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners (pp. 59–64). Apress. https://doi.org/10.1007/978-1-4842-4470-8_7
4. Coombes, M., Eaton, W., & Chen, W. H. (2017). Machine Vision for UAS Ground Operations: Using Semantic Segmentation with a Bayesian Network classifier. Journal of Intelligent and Robotic Systems: Theory and Applications, 88(2–4), 527–546. https://doi.org/10.1007/s10846-017-0542-5
5. Cremona, J., Uzal, L., & Pire, T. (2022). WGANVO: monocular visual odometry based on generative adversarial networks. RIAI - Revista Iberoamericana de Automatica e Informatica Industrial, 19(2), 144–153. https://doi.org/10.4995/RIAI.2022.16113
6. Dhulipudi, D. P., & Rajan, K. S. (2020). Multiclass geospatial object detection using machine learning-aviation case study. AIAA/IEEE Digital Avionics Systems Conference - Proceedings, 2020-Octob. https://doi.org/10.1109/DASC50938.2020.9256771
7. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Annual Conference on Robot Learning, 78, 1–16. https://proceedings.mlr.press/v78/dosovitskiy17a.html
8. Ducoffe, M., Carrere, M., Féliers, L., Gauffriau, A., Mussot, V., Pagetti, C., & Sammour, T. (2023). LARD – Landing Approach Runway Detection – Dataset for Vision Based Landing. http://arxiv.org/abs/2304.09938
9. European RPAS Steering Group, Tech. Rep. (2013). Roadmap for the integration of civil Remotely-Piloted Aircraft Systems into the European Aviation System.
10. Jocher, G., Chaurasia, A., & Qiu, J. (2023). Ultralytics YOLO. https://github.com/ultralytics/ultralytics
11. Lee, S., & Seo, S. W. (2022). Probabilistic context integration-based aircraft behaviour intention classification at airport ramps. IET Intelligent Transport Systems, 16(6), 725–738. https://doi.org/10.1049/itr2.12167
12. Lu, B., Coombes, M., Li, B., & Chen, W. H. (2016). Improved situation awareness for autonomous taxiing through self-learning. IEEE Transactions on Intelligent Transportation Systems, 17(12), 3553–3564. https://doi.org/10.1109/TITS.2016.2557588
13. OneView, & Airbus Defence and Space Intelligence. (2021). Can Synthetic Data Really Improve Algorithm Accuracy? https://ai4.io/pdfs/OneView-Airbus-Ai4-Press-Release.pdf
14. Rong, G., Shin, B. H., Tabatabaee, H., Lu, Q., Lemke, S., Mozeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., Agafonov, E., Kim, T. H., Sterner, E., Ushiroda, K., Reyes, M., Zelenkovsky, D., & Kim, S. (2020). LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 1–6. https://doi.org/10.1109/ITSC45102.2020.9294422
15. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3234–3243. https://doi.org/10.1109/CVPR.2016.352
16. Shaikh, M. Y., Petrunin, I., & Zolotas, A. (2023, March). Self-supervised Obstacle Detection During Autonomous UAS Taxi Operations. AIAA SCITECH 2023 Forum. https://doi.org/10.2514/6.2023-2672
17. Shermeyer, J., Hossler, T., Etten, A. van, Hogan, D., Lewis, R., & Kim, D. (2021). RarePlanes: Synthetic Data Takes Flight. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 207–217. https://doi.org/10.1109/WACV48630.2021.00025
18. Steininger, D., Kriegler, A., Pointner, W., Widhalm, V., Simon, J., & Zendel, O. (2023). Towards Scene Understanding for Autonomous Operations on Airport Aprons (pp. 153–169). https://doi.org/10.1007/978-3-031-27066-6_11
19. Studio, A., & Studios, X. G. (2020). Microsoft Flight Simulator 2020. https://flightsimulator.com/
20. Suder, J., Podbucki, K., Marciniak, T., & Dabrowski, A. (2021). Low complexity lane detection methods for light photometry system. Electronics (Switzerland), 10(14). https://doi.org/10.3390/electronics10141665
21. Talwar, D., Guruswamy, S., Ravipati, N., & Eirinaki, M. (2020). Evaluating Validity of Synthetic Data in Perception Tasks for Autonomous Vehicles. Proceedings - 2020 IEEE International Conference on Artificial Intelligence Testing, AITest 2020, 73–80. https://doi.org/10.1109/AITEST49225.2020.00018
22. Tomaszewska, J., Krzysiak, P., Zieja, M., & Woch, M. (2018). STATISTICAL ANALYSIS OF GROUND-RELATED INCIDENTS AT AIRPORTS. https://doi.org/10.5604/01.3001.0012.4369
23. Zhang, Y., Poupart-Lafarge, G., Teng, H., Wilhelm, J., Jeannin, J.-B., Ozay, N., & Scholte, E. (2020, March). A software architecture for autonomous taxiing of aircraft. AIAA Scitech 2020 Forum. https://doi.org/10.2514/6.2020-0139

---

By Miguel Ángel de Frutos Carro, Carlos Cerdán Villalonga & Antonio Barrientos Cruz - 2024