Using Docker for replication
Install docker for your environment. We have tested on docker version 18.09.1, build 4c52b90 on Ubuntu 18.04. One may also follow the official install instructions from here: https://docs.docker.com/install
Download some required data from this link, unzip and place in the home directory. We refer this unzipped folder as the data folder throughout this documentation. You may also put the data directory anywhere but while running the docker containers, you need to provide the absolute path of this directory. The following instructions are written with the assumption that the unzipped data folder is placed in the home directory. Please adapt the absolute path of the data folder according to your environment and USER_NAME.
The template command for running the containers is
docker pull jibesh/nl2type:TAG docker run -v PATH_TO_DATA_DIR:/data jibesh/nl2type:TAG
PS: You might need to prepend sudo to each of the following commands
Download the containers
- To replicate the results from Table 1 of our paper, run the following commands:
# Pull the container docker pull jibesh/nl2type:table1 # Execute the script docker run -v /home/USER_NAME/data:/data jibesh/nl2type:table1
Results are printed on the terminal which corresponds to the first row of Table 1 (Approach: NL2Type) of our paper. The final output of this command is placed in data/paper/results/results.csv
- To replicate the results for the model trained only on names and not on the comments is
docker pull jibesh/nl2type:table1_no_comments docker run -v /home/USER_NAME/data:/data jibesh/nl2type:table1_no_comments
Results are printed on the terminal which corresponds to the second row of Table 1 (Approach: NL2Type w/o comments) of our paper. The final output of this command is placed in data/paper/results/predictions_paper_no_comments.csv
- To use the model to make predictions using the same test data as used in the paper, run the following commands. The test data is placed in data/paper/raw_csv/test.csv
docker pull jibesh/nl2type:from_vecs docker run -v /home/USER_NAME/data:/data jibesh/nl2type:from_vecs
The final output of this command is placed in data/paper/results_new_enriched.csv. This script first vectorizes the data in this test file and uses the same model as used in the paper to make predictions. This corresponds to steps 3, 4 and 5 in Figure 2 in the paper.
docker pull jibesh/nl2type:demo docker run -v /home/USER_NAME/data:/data jibesh/nl2type:demo
The predictions for the given files will be data/demo/results/results.csv
If you do not want to use docker please use the following setup setup and instructions to run our tool.
Requirements and assumptions
- python 2.7
- pip2 (Tested using version 9.0.1 for python 2)
- Tested on Ubuntu 18.04.1 LTS
Download some results and required data from this link, place it in current directory and unzip it.
To download the files used for training and testing the model used in the paper, use this link. The files used for training the model are in "training_files" and the files used in testing are in "testing_files"
- Install the dependencies using the following command
pip2 install --upgrade -r requirements.txt
- Install Node.js and also download a required Node.js package using the following command
sudo apt-get install -y nodejs npm install -g jsdoc
Results and replication
The model used in the paper is in models/model.h5
The main results file from the paper is data/paper/results/results.csv. The following commands calculate the results in Table 1 in the paper from this results file:
python2 scripts/runner.py --config scripts/configs/stats_paper.json
The results for the model trained only on names and not on comments is data/paper/results/predictions_paper_no_comments.csv. The following commands calculate the results for no comments in Table 1 in the paper from this file:
python2 scripts/runner.py --config scripts/configs/stats_paper_no_comments.json
In the results file, the column "original" contains the actual type of the datapoint, the column "top_5_prediction" refer to the top 5 most likely predictions as explained in the paper, separated by the token "%". The column "datapoint_type" indicated whether the point is a function or parameter, with the value 0 for a function and 1 for a parameter.
- To use the model to make predictions using the same test data as used in the paper, run the following command:
python2 scripts/runner.py --config scripts/configs/from_vecs.json
This makes predictions for the points present in the file data/paper/raw_csv/test.csv and the generated results file will be data/paper/results_new_enriched.csv
- The results from the inconsistency analysis are in data/paper/results/inconsistency_analysis_paper.csv
python2 scripts/runner.py --config scripts/configs/demo.json