Artifact for the OOPSLA 17 DejaVu Paper
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


title output
Javascript Pipeline

The entire JavaScript pipeline is outlined in this readme file with links to detailed descriptions of respective sections. The workflow assumes that all commands are executed from the directory where the notebooks are stored so that the notebooks can be executed from within R studio. For every file, double click it, and then press CTRL+ALT+R to execute the code.

For more information about R notebooks in R Studio, see R Notebooks in R Studio.

What to do:

  • if you are evaluation the VM artifact, skip steps 0 and 1 as they were already executed (using the dataset sample) and jump immediately to step 2.
  • if you are evaluating on the entire dataset on our server using the datasets provided there, you can only execute steps 8 and 9.
  • if you want to reproduce the entire pipeline, start with step 0. Note that redoing the entire pipeline on entire github will take weeks.

When time estimates are provided in the steps they usually concern the VM running time on the dataset sample. Running time on full dataset is in days and weeks.

0 - Setup & Tools

This step has already been executed on the VM

An overview of the required tools and their setup.

Detailed Information can be found in file 0-setup.Rmd.

1 - Getting Github project urls

This step has already been executed on the VM

In this step, GHTorrent database snapshot is used to filter out list of available project urls on github, forks, deleted and duplicated projects are excluded.

Detailed Information can be found in file 1-getting-projects.Rmd.

2 - Downloading and tokenizing the projects

This step details the download & tokenization techniques for the filtered projects. After this step the file & token hashes for all files are known.

Detailed Information in file 2-tokenizing.Rmd.

3 - SourcererCC Clone Analysis

Tokenized files are fed into the SourcererCC and similar files are found.

Detailed Information in file 3-sourcerercc.Rmd.

4 - Database Import

The computed data is loaded into a MySQL database

Detailed Information in file 4-dbimport.Rmd.

5 - Project Clones

Clone information for entire projects is calculated (this is the input data for the heatmaps).

Detailed Information in file 5-project-clones.Rmd.

6 - Metadata Acquisition

Dowloads metadata for the projects directy from github and from the ghtorrent snapshot.

Detailed Information in file 6-metadata.Rmd.

7 - Additional Processing

Additional processing of inputs to the reporting steps using both files on disk and the database itself.

Detailed Information in file 7-processing.Rmd.

8 - Data Analysis & Reporting

This step can be executed on the machine with all data loaded to reproduce the results of the paper.

These steps produces the graphs and tables in the paper. The graphs and tables in this section aggregate information for all four languages, but the notebook prints only JavaScript part for now. However, you can easily change the dataset used in the config.R to view reports of other languages as well.

Detailed Information in file 8-reports.Rmd.

9 - Language specific reporting

This step can be executed on the machine with all data loaded to reproduce the results of the paper.

These graphs from the paper apply only to the JavaScript since they either have no meaning (npm vs non-npm distinction), or the other languages do not provide the additional information required to construct them (time aggregation).

Detailed Information in file 9-reporting-other.Rmd.