Identifying Safety-Critical Concerns in Unmanned Aerial Vehicle (UAV) Software Platforms with SALIENT

SALIENT is a machine learning (ML)-enabled tool which analyzes individual sentences composing the issue reports of UAVs and automatically recognizes those describing a safety-related concern. The assessment of the classification performance of the tool on the issues of three relevant open-source UAV-related projects demonstrates that SALIENT represents a viable solution to assist developers in timely identifying and triaging safety-critical UAV issues in GitHub.

Table of Contents

• Architecture Overview
• HOW TO USE SALIENT
• Setup
  • Local Development Environment with GUI
  • Using Docker
• Command Line Interface
• Graphical User Interface
• License
• References
• Contacts

Architecture Overview

As shown in the Figure below, SALIENT supports two main usage modes thanks to its CLI and its GUI. The GUI is designed for general users (not necessarily developers), while the CLI is useful for UAV developers interested in automating the issue management process toward safety-related issues. Both the CLI and GUI are designed to allow developers/users to select issues as well as write and edit issues. Then, specific CLI commands and specific GUI events support the developers/users in the automated identification (or classification) of sentences in issues concerning safety-related aspects of UAVs. The CLI and the GUI of SALIENT enable the aforementioned steps by leveraging (or interacting with) two components: the GitHub Issue Parser and the ML Component. The users/developers (i) interact with the GitHub Issue Parser via the CLI or the GUI, with the component able to preprocess issues downloaded from Github in different formats (explained in later sections) and preprocess them (e.g., splitting the issue text logically in sentences) for the next analysis. Once the data are pre-processed, (ii) the users/developers interact with the ML Component via the CLI (or the GUI) of SALIENT. which allows them to identify sentences with safety-related implications. Specifically, in this process, the data pre-processed by the GitHub Issue Parser are then provided as input to the SALIENT ’s ML Component which identifies (or classifies) the individual sentences that are safety-related. To support developers/users during the issue management process, the ML Component uses a model we trained in our previous work (reference mentioned in this repository) to classify safety-related sentences in newly reported UAV issues in GitHub. The training, tuning, and accuracy of the ML model employed by SALIENT are detailed in our previous work (reference mentioned in this repository).

HOW TO USE SALIENT

Demonstration Video (short version - around 5 minutes):

• Youtube link

Demonstration Video (long version - over 10 minutes):

• Youtube link

Setup

1. Clone this repository

git clone https://github.com/spanichella/SALIENT-TOOL.git

Local Development Environment (With GUI)

2. Make sure you have all the Requirements

• Python 3.9
• Docker
• Poetry
• tkinter
• pandas
• numpy

apt-get update
sudo apt-get install tk # on macosx pip3 install tk

3. Install the requirements usint poetry:

poetry install
cat requirements.txt | xargs -I % sh -c 'poetry add "%"'
pip install -r requirements.txt

4. Install fasttext:

pip install fasttext==0.9.1

5. cd salient_src

6. Open the GUI:

python salient_gui_tkinter.py # before running this command it could be necessary to open a new terminal (we are assuming that after installing required tools/packages, the running of the GUI requires to run an independent command line to have the installation process be effective)

7. Run the CLI:

python Fasttext-Model-prediction-on-safety-unseen-data.py --infile config.json

Using Docker

2. Build and tag the docker image:

docker build . -t salient_tool # in macosx can require "sudo": sudo docker build . -t salient_tool

3. Run the container and open bash:

docker run -it salient_tool bash

4. Start using the CLI:

python Fasttext-Model-prediction-on-safety-unseen-data.py --infile config.json

(Optional) In case a container with the same name is already running and you want to remove it (i.e., stop container):

docker container ls
docker container stop $(docker container ls -aq)	
docker system prune
docker container ls

Command Line Interface

SALIENT can be used as a Python command-line utility, which needs proper setup and configurations for execution. To simplify this process, we have included a Dockerfile that sets up all the requirements, runs the container, and opens the bash to interact with the tools, as reported in previous steps. To invoke SALIENT within (and outside) the container, with the configuration file config.json, SALIENT provides the following interfaces:

1. python Fasttext-Model-prediction-on-safety-unseen-data.py --infile config.json

As demonstrated in the video and detailed in our repository, SALIENT’s config.json file contains different arguments as summarized in Table II (see figure below), which allows specifying the required inputs (i.e., the issue data), with the command above, to proceed with the analysis/classification of issues data and store the results in the specified file (with the file config.json). It is important to note that, if multiple Input types (see Table II) are provided as input with the config.json file, all of them are analyzed/classified and stored in the specified output file.

The CLI also allows storing the results locally. SALIENT provides different options to output the results with its GUI and its CLI. It is important to note that given as input issue data, the output of SALIENT corresponds to a structured file (in CSV format) containing the sentence ID, the text of the sentence classified, the label assigned to the sentence ("Yes" or "No", depending on the fact that the sentence concerns safety aspects), and the probability that the predicted label is actually correct:

Sentence,Predicted_Label,Probability_of_Predicted_Label
0,AP_Arming: pre-arm check if compass1 is disabled but 2 or 3 are enabled.,"('__label__YES',)",0.670190691947937
...

Graphical User Interface

Users can access the GUI with the following command:python salient_gui_tkinter.py

As described before, to activate the GUI, it is sufficient to run python salient_gui_tkinter.py The GUI is designed for general users (not necessarily developers) to allow developers/users to select issues as well as write and edit issues. As demonstrated in the video (before in this repository), the SALIENT’s GUI allows to upload a text file (or CSV file) containing the text of the UAV issue (point 1in the figure), or the users/developers can directly copy and paste the text of the issue (point 2) in the GUI’s form (see image before).

With the GUI it is possible to classify text of issues concerning safety aspects automatically (point 3), with relevant sentences highlighted in the specified color (default it is orange color). The GUI also allows storing the results locally. As described in previous paragraphs, SALIENT provides different options to output the results with its GUI (point 4) and its CLI (see previous paragraphs). It is important to note that given as input issue data, the output of SALIENT corresponds to a structured file (in CSV format) containing the sentence ID, the text of the sentence classified, the label assigned to the sentence ("Yes" or "No", depending on the fact that the sentence concerns safety aspects), and the probability that the predicted label is actually correct:

Sentence,Predicted_Label,Probability_of_Predicted_Label
0,AP_Arming: pre-arm check if compass1 is disabled but 2 or 3 are enabled.,"('__label__YES',)",0.670190691947937
...

License

SALIENT tool for the automated identification of Safety Concerns Reported in UAV Software Platforms.
Copyright (C) 2022  Sebastiano Panichella

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.

The software we developed is distributed under GNU GPL license. See the LICENSE file.

References

If you use this tool in your research, please cite the following paper(s):

• Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms. Transactions on Software Engineering and Methodology. 2022.

@article{UAV:2022,
   author = {Di Sorbo, Andrea and Zampetti, Fiorella and Visaggio, Aaron and Di Penta, Massimiliano and Panichella, Sebastiano},
   title = {Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms},
   year = {2023},
   publisher = {Association for Computing Machinery},
   volume = {32},
   number = {3},
   doi = {10.1145/3564821},
   journal = {ACM Trans. Softw. Eng. Methodol.},
   articleno = {67},
   numpages = {37}
}

• F. Zampetti, R. Kapur, M. Di Penta, and S. Panichella, “An empirical characterization of software bugs in open-source cyber–physical systems,” Journal of Systems and Software, vol. 192, p. 111425, 2022.

@article{ZAMPETTI2022111425,
  title = {An empirical characterization of software bugs in open-source Cyber–Physical Systems},
  journal = {Journal of Systems and Software},
  volume = {192},
  pages = {111425},
  year = {2022},
  issn = {0164-1212},
  doi = {https://doi.org/10.1016/j.jss.2022.111425},
  url = {https://www.sciencedirect.com/science/article/pii/S0164121222001315},
  author = {Fiorella Zampetti and Ritu Kapur and Massimiliano {Di Penta} and Sebastiano Panichella}
}

Contacts

• Dr. Sebastiano Panichella
  • Zurich University of Applied Sciences (ZHAW), Switzerland - panc@zhaw.ch and spanichella@gmail.com
• Sajad Khatiri
  • Zurich University of Applied Sciences (ZHAW), Switzerland - mazr@zhaw.ch
• Dr. Andrea Di Sorbo:
  • University of Sannio, Italy - disorbo@unisannio.it