Benchmarks

jabiercoding edited this page Mar 20, 2017 · 7 revisions

Eclipse Feature Location Benchmark for Software Families uses variants of Eclipse from which a ground-truth is extracted.

A set of Feature Location techniques have been already integrated in BUT4Reuse.

Do you have your own feature location technique?

Do you want to compare it with others in this benchmark?

Instructions on integrating your feature location technique to be used within EFLBench:

Cite as: Jabier Martinez, Tewfik Ziadi, Mike Papadakis, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon : Feature Location Benchmark for Software Families using Eclipse Community Releases, ICSR 2016, Limassol, Cyprus, 5-7 June

Benchmark construction

Get the Eclipse package variants

Select the distribution from which you want to create the benchmark: http://www.eclipse.org/downloads/packages/all

For example, you can use Kepler SR2, download the win 64 bits. http://www.eclipse.org/downloads/packages/release/Kepler/SR2

Notice that it does not matter that you are not using windows nor 64 bits. You will not need to execute it and EFLBench automatic extractor will not need to execute them to prepare the feature location benchmark.

Download and Unzip all. This process can take some time depending on your internet connection and computer. At the end, you should have a folder like the one in the image with the 12 packages of the Kepler SR2 release

Install BUT4Reuse and prepare EFLBench

Install BUT4Reuse and launch it. For this follow the instructions at [Installation](Installation)

Create a project and an artefact model. For example, call the project packagesKepler and call kepler to the artefactmodel. Then open the kepler.artefactmodel in the artefactmodel editor by double clicking it. Select the root and in the properties view, put kepler as name. For these steps you can find further information in the BUT4Reuse user manual: UserManual

Be sure that the kepler.artefactmodel is still open in the editor.

Drag and Drop the unzipped folders of the eclipse distributions in the BUT4Reuse's Input Drop view.

Select each of the artefacts and in the Properties view, be sure that the URI is pointing to the eclipse folder. Most probably you will need to add eclipse/ manually. For example file:/C:/keplerPackages/eclipse-jee-kepler-SR1-win32-x86_64/eclipse/

Launch EFLBench automatic extractor

Right click the artefact model and click on "Create Eclipse Feature Location Benchmark" and wait.

This action can take around 5 minutes depending on your computer. Once it is finished a summary is shown.

A benchmark folder is created

actualFeatures folder: This contains the ground truth. Inside the folder you will find a txt file for each feature and in these txt files you have, in each line, the ids of the plugins that constitutes this feature.

eclipse_kepler.featurelist: This contains the information to launch feature location techniques. You can open it with the featurelist editor. It is a feature list with feature names and descriptions. In the Eclipse properties view, by selecting a feature you can see in which artefacts this feature is implemented.

Then we have two graphml files with mined information that can be exploited by feature location techniques. You can use Gephi for visualising graphml files: https://gephi.org/ . If you want to include a feature location technique that leverages this information you can use for example the blueprints library, com.tinkerpop.blueprints.Graph. However, plugin element dependencies to other plugin elements can be obtained through the getDependencies method of the Element interface

actualFeatures.graphml: a graph containing feature dependencies

pluginDependencies.graphml: a graph containing plugin dependencies

The following figure shows the feature list with Eclipse CVS Client feature selected and their properties name, description and in which packages it is present.

Benchmark usage

To change the feature location techniques go to Window -> Preferences, and select BUT4Reuse

Here you can find the feature location techniques registered.

In the preferences, in Block creation, you can select for example Formal Concept Analysis (select only one).

Then select also only one of the feature location techniques, for example SFS

In order to reduce the time that you will have to wait until getting the results, in the preferences, in Visualisation, only check Metrics and Eclipse benchmark specific visualisation. Also in Constraints discovery, do not select any of the techniques.

Metrics visualisation will provide us the information about the time performance and the other will provide the precision and recall results.

Right click the root of the Feature list and click on Feature location

They will ask you for the adapter to use, the abstraction phase for the variants. Select only the Eclipse4Benchmark adapter

A csv file (comma-separated-value) called resultsPrecisionRecall_techniqueSelected_timestamp will be created containing the precision and recall for each feature

The time performance is shown in the Metrics view. It is at the end of metrics report.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.