OpenScienceStudy

Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability Detection

Original artifact	https://figshare.com/s/e048fa191503393275a1
Imported from	the publications page
Tool	pubs2github

---

Contents

The artifact contains 936 file(s) including Python, Java, C/C++, Shell scripts, Config files, Data files, and Documentation.

├── 9_Projects
│   └── .DS_Store
├── DCKM
│   ├── DCKM-replicability
│   │   ├── __pycache__
│   │   ├── converter
│   │   ├── datasets
│   │   ├── graphs
│   │   ├── log
│   │   ├── results
│   │   ├── saved-model
│   │   ├── .main.py.un~
│   │   ├── .utils.py.un~
│   │   ├── main.py
│   │   ├── main.py~
│   │   ├── main_balance10pct.py
│   │   ├── main_balance4pct.py
│   │   ├── main_balance7pct.py
│   │   ├── main_partial_repli.py
│   │   ├── main_size10.py
│   │   ├── main_size40.py
│   │   ├── main_size70.py
│   │   ├── README.md
│   │   ├── utils.py
│   │   └── utils.py~
│   └── DCKM-reproducibility
│       ├── __pycache__
│       ├── datasets
│       ├── graphs
│       ├── log
│       ├── saved-model
│       ├── LICENSE
│       ├── main.py
│       ├── README.md
│       ├── results.txt
│       └── utils.py
├── LAVDNN
│   ├── LAVDNN-replicability
│   │   ├── Code
│   │   ├── Data
│   │   ├── Model
│   │   └── README.md
│   └── LAVDNN-reproducibility
│       ├── Code
│       ├── Data
│       ├── Model
│       └── README.md
├── MDSeqVAE
│   ├── MDSeqVAE-replicability
│   │   ├── __pycache__
│   │   ├── dataset
│   │   ├── dataset10
│   │   ├── dataset40
│   │   ├── dataset70
│   │   ├── dataset_10pct
│   │   ├── dataset_4pct
│   │   ├── dataset_7pct
│   │   ├── graphs
│   │   ├── results
│   │   ├── saved-model
│   │   … (122 more items)
│   … (269 more items)
… (1326 more items)

---

Original README.md (from the upstream artifact)

On the Open Science of DL-Based Software Vulnerability Detection Techniques

Replicating and reproducing available tools to investigate the status quo of open science in the topic of DL-based software vulnerability detection techniques.

We investigated 56 papers/studies. 15 papers/studies provided available tools. In the 15 available tools, we excluded tools introduced in posters and targeting languages except C/C++ source code and compiled C/C++ binary code. Thus, we tried to execute 11 tools. However, only 8 tools were executable. In this package, we only provide the eight executable tools.

Package Structure

• RQs/: The complete raw data of the four research questions.
• "Tool_Name"/: The source code of the tools we used for our reproducibility and replicability experiments.
  • *-reproduciblity/: The source code for our reproducibility experiments. -README.md: The documentation for setting up the tool, executing the reproducibility experiment, and finding the experiment results we have gotten.
  • *-replicability/: The source code for our replicability experiments. -README.md: The documentation for setting up the tool, preprocessing third-party program samples, executing the replicability experiment, and finding the experiment results we have gotten.
• 9_Projects: The dataset we used for our replicability experiments.

How to use

To review our experiment results of the four RQs, please check the spreadsheet in the "RQs" folder.

To replicate our replicability/reproducibility experiments of a tool, go to the tool folder and follow the instructions in its "README.md" to set up the tools, preprocess the data, and execute the experiments.

To find the raw outputs of our experiments, go to the tool folder and read its "README.md" file to find the locations of the output files.