Cruz_TracelabSummaries

[mcmillco]

No description provided.

[hongyujohn]

We encounter some problems in evaluating this tool: how to summarize natural language descriptions from raw code?  It seems that some key components such as SWUM are missing. There is no instruction how to produce summarizations from source code using the provided 18 components.

[mcmillco]

Hi there! Perhaps I can answer those two questions: First, SWUM is a research prototype owned by the University of Delaware. They were kind enough to provide it to us so we could use it as a component in our research. Unfortunately we don't have the rights to redistribute it by itself, so we can't include it separately. But we can distribute it as packaged within our work with all due credit, so if you use the VM you should be able to run everything just fine. If you want to run it outside the VM, it is just a matter of asking the UDel team (Lori Pollock or Emily Hill who was formerly with that team), and dropping the files they send you into the ND_Sumslice directory. I am sorry that step is a bit complicated, but there is no legal way around it right now. If there are other components you found to be missing, please let me know as I am not aware of any.

Second, actually we have quite thorough instructions. Please follow the URL in the scorecard and paper:
http://www.cse.nd.edu/~cmc/tracelab/icsme16/
If you click on any of the components, you will find step-by-step instructions including screenshots. We made the screenshots from the VMs, so if you use those you can pretty much just follow along.

I hope this clears up those issues. Please let me know if not! Just FYI, I am traveling through a remote area with unreliable internet, so may not be able to respond until Sunday evening or Monday morning.

Collin

[yasutaka-kamei]

Dear @cmcmil

I merged the divided zip files by cat ubu.z01 ubu.z02 ubu.z03 ubu.zip > output.zip and unzip output.zip, then obtained the following error:

[ubu]$ unzip output.zip 
Archive:  output.zip
warning [output.zip]:  zipfile claims to be last disk of a multi-part archive;
  attempting to process anyway, assuming all parts have been concatenated
  together in order.  Expect "errors" and warnings...true multi-part support
  doesn't exist yet (coming soon).
warning [output.zip]:  12884901964 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [output.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

I also tried cat ubu.zip ubu.z01 ubu.z02 ubu.z03 > output1.zip, but it did not work too. Would you please tell me what the problem could be? I share md5 information in the case of what my downloaded files are broken.

[ubu]$ openssl md5 ubu.z01 
MD5(ubu.z01)= b86235a59911b2508b7e0895bed7e0a2
[ubu]$ openssl md5 ubu.z02
MD5(ubu.z02)= 4b1ad5a80c455c2d4bfeb8b0d58344e6
[ubu]$ openssl md5 ubu.z03
MD5(ubu.z03)= 4f044791ab69a4083c11df04c8325bb9
[ubu]$ openssl md5 ubu.zip 
MD5(ubu.zip)= 5f1e79e85ae09a584a6dbf52e78e8034

[mcmillco]

I used this command to create one large archive out of the pieces:

zip -F ubu.zip --out ubu_full.zip

and then this command to unzip the large archive:

unzip ubu_full.zip

Sorry for the confusion about that and thanks for pointing it out. We can include a note on the webpage to make it easier for everyone.

[hongyujohn]

We still cannot see any summary results. how can we produce the /home/everyman/TraceLab/Data/NanoXML.out files? what programs/tools to use?

[mcmillco]

Great question. To produce the /home/everyman/TraceLab/Data/NanoXML.out files (the files containing the summaries for NanoXML), you need to run the Summary Generation experiment in TraceLab. All you need to do is right click and go to terminal (xterm is fine) and run tracelab. Then open and run the experiment file (it's under /home/everyman/TraceLab/Experiments).

We provide a step-by-step instruction guide on our website. It includes screen shots to make it easier:
http://www.cse.nd.edu/~cmc/tracelab/icsme16/sample_exp2.htm

[yasutaka-kamei]

@cmcmil

Thanks for the information, but I still face another error message.

[ubu]$ zip -F ubu.zip --out ubu_full.zip
Fix archive (-F) - assume mostly intact archive
 copying: ubu/
 copying: ubu/ubu.ova
[ubu]$ unzip ubu_full.zip
Archive:  ubu_full.zip
warning [ubu_full.zip]:  4294967369 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [ubu_full.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

[mcmillco]

@yasutaka-kamei Let me look into it. I wonder if it is a version problem. I'll get back ASAP.

[mcmillco]

@yasutaka-kamei By the way, are you able to access the Debian virtual machine?

[mcmillco]

Just to continue updating you on the status: I am trying to get in touch with Breno, who created the Ubuntu VM. In the meantime, I wonder if you might try the experiment in the Debian VM, which I built.

Best,
Collin

[yasutaka-kamei]

@cmcmil

Yeah, the Debian VM works for me! How do I use the Debian VM for the experiment, because it seems to provide only a CUI interface unlike the interface a step-by-step instruction guide shows?

[mcmillco]

@yasutaka-kamei

The Debian VM works great for the Summary Generation Experiment:
http://www.cse.nd.edu/~cmc/tracelab/icsme16/sample_exp2.htm

In fact, the screenshots on that page are made with the Debian VM. :-)

All you need to do is log in (everyman / password) and type "startx" to start the OS GUI (sorry it doesn't come up automatically, we thought this way would increase compatibility).

Once in the GUI, just right click and select xterm. Then type "tracelab" to start TraceLab. If you get an "object reference" error, just try to open TraceLab again after a couple seconds. That is a bug in TraceLab which we unfortunately can't do anything about until the developers fix that. Once it opens everything works fine.

[yasutaka-kamei]

@cmcmil

Great! It works for me now! Thank you so much for the information.

[mcmillco]

Great. Let me know if anything else comes up!

I have not modified the webpage yet in order to avoid circumventing the deadline, but once the notifications go out, I will add clarifications about all of this.

[yasutaka-kamei]

This artifact provides a reproducibility package for experiments in source code summarization. The artifact generates natural language text that describes the behavior of the given source code.

Thank you for answering my previous questions. I have the following comments that may help to improve the paper and supplemental web page:

• As the paper mentions, the artifact is implemented as a set of components for the TraceLab research infrastructure. However, the paper does not mention what TraceLab components are.
• I downloaded a Debian image first because it was smaller than an Ubuntu image, but could not understand how to use the Debian one. So please revise the web page to add the information the authors gave me previously.
  (@cmcmil has already mentioned that they would add it later :-) )
• At the page (http://www.cse.nd.edu/~cmc/tracelab/icsme16/sample_exp2.htm), please add more steps about how to go to the picture at step 1 (e.g., Click Open Existing Experiment and Choose full.teml).
• I would like to encourage the authors to have (1) how to run experiments using another project, not the NanoXML project and (2) how to add new components that are not registered yet in the supplemental web page.
• How do I interpret the output of full.teml? The following output seems not to be Source code summarization.

Messages: 2082
Method Name: equalsXMLElement
-> QuickSummaryMessage -> ReturnMessage -> OutputUsedMessage ->

The artifact reproduces source code summarization. Source code summarization is a new research area, so I think that the artifact is insightful, useful, and usable. I'm happy to accept this paper.

[hongyujohn]

@cmcmil thanks for your explanation. As for the missing SWUM component, you mentioned : "But we can distribute it as packaged within our work with all due credit, so if you use the VM you should be able to run everything just fine. " Is it already included into the current package?

[emhill]

If we desperately need the prototype SWUM eclipse-plugin for Java to evaluate the tool, I'm happy to have it shared here (and would consider a license change). We have a programming-language independent version (https://github.com/abb-iss/Swum.NET), but I doubt that's what is being used here.

[mcmillco]

@emhill Thanks Emily!! To the question from @hongyujohn, SWUM is packaged as part of the component inside the VM, so everything should work there without modifications. But we do not provide SWUM as a standalone download, and do not provide any instructions for separating SWUM from our component. I suppose it could be decompiled, but it would be way easier to just ask Emily to send it to you. :-P

The PL-independent version came out after we built our original tool, so we still use the older one. I highly recommend that future development use the new version.

[hongyujohn]

thanks all for your clarifications! I will upload my review tomorrow.

[brenodan]

Hello all,

Sorry for the absence until the present moment.
I merged the Ubuntu VM and updated the web site (http://www.cse.nd.edu/~cmc/tracelab/icsme16/index.htm).
Please feel free to try it.

[mcmillco]

Just to clarify, the link to download the merged image file for the Ubuntu VM is here:
https://notredame.box.com/s/zeurku4v4uz1up178pxsdbbnkkdz3dso

We will update the web site once the notification deadline passes.

[hongyujohn]

Summary
This artifact is a reproducibility package for source code summarization. It is implemented as a set of components for the TraceLab research infrastructure. We are able to run this tool (although some third-party components are missing as explained by the authors). I would recommend acceptance.

Insightful
Source code summarization is a useful task in software engineering. The tool could reduce the time for researchers reproducing code summarization tasks. The artifact could be used for other related experiments on source code summarization.

Useful
The artifact can be used to reproduce some of the authors’ experiments. However, some results (i.e., the SWUM results) are provided by the authors but the SWUM component is not provided. I understand that this component is not for general public due to copyright concern, but it should be provided to the reviewers for examination. This may also affect the practical use of the tool.

Usable
The configuration is a bit complex. Users need to go through many internal I/O configurations. Why not provide a complete end-end package that allows users to directly see summarizations given some code snippets?

[ghost]

This artifact consists of a package for experiments conducted in the area code summarization. The artifact is provided in the form of components for the TraceLab Infrastructure.

Recommendation
‘? accept’.

Insightful

The idea of the paper is interesting, timely, and relevant for this ICSME 2016 artifact track.

The paper is well motivated and the problem is clearly stated.

Useful

Code element summarization is not a trivial task and hence this artifact can be very useful for researchers working on this area. It can help them reproduce existing works and—or enhance previous techniques.

Another advantage in favor of the paper is that the techniques reproduced by this artifact have been already published.

Usable

The authors adhere to the ICSME artifact track guidelines and particularly the three required dimensions: insightfulness, usefulness, and usability.

Even given the high number of components in TraceLab, which makes it not that much easy to reproduce experiments, the authors have deployed efforts in explaining and providing details about their package.

Overall, I believe this artifact can be of great value to the SE community working on natural language summarization of code elements.

[obaysal]

In spite of the complex configuration of the artifact, all reviewers found it useful and recommended "accept". I'll mark is accordingly.