Replication Package of our paper "How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review"
A pre-print version can be found at: https://arxiv.org/abs/2107.12045
The file is a .xls file, with each sheet representing one step of our methodology:
-
Raw Data: Sheet containing all the data after Pooling and Duplicate Filtering (Section 3.4.2 in the paper). We reported Title, Author, URL, Year extracted from the different databases (DB).
-
GS Round 2: Sheet containing our decision for the step of Second Round of Google Scholar Pruning (Section 3.4.3 in the paper). The decision process is in two steps, done by two reviewers that inspect title and abstract of papers for relevance to our study, that is if it's not out of our scope: First round (column Pass/Fail) prunes papers by rejecting papers with two rejects (in red in the sheet) or accepting papers with two accepts (in green in the sheet). Reviewers' disagreement on any paper (in orange in the sheet) lead to a discussion between reviewers, resulting in the second round (Decision Post Discussion) where the final decision is taken.
-
Inclusion & Exclusion: Sheet containing our decision for the step Inclusion and Exclusion criteria (Section 3.4.4 in the paper). The decision is made in two steps, just like the previous step. Here, the Inclusion/Exclusion criteria are evaluated as presented in our paper.
-
Main sheet: The main sheet regroups the papers considered for Quality Control Assessment (from Section 3.5 in our paper). We provide information used in the Data Extraction part (Section 3.6), as well as data used for figures in Section 3 (venue, type of the venue, model type used ...). Papers highlighted in red were rejected before Quality Control because they were out of our scope. Papers highlighted in blue were rejected for having a too low score during Quality Control evaluation. Note that we also count the 4 papers of the snowballing process.
-
Control Question Score: This sheet regroups the quality control scoring registered by two different reviewers. We summed the score obtained for questions Q1 to Q7, as well as the score for question Q8 and Q9a/Q9b as described in the paper. Then, the average is taken. To be selected, a paper needs to have a score of at least 7 for questions Q1-Q7, and 1 for either Q8 or Q9a/Q9b. Papers in green were accepted and papers in red rejected.