processing dirk's comments on artifact predicates

[timm]

Conference Publishing - Info
2:01 PM (9 hours ago)

to me, Tom, Thomas, Tim, Olga

do you have any more info on the ACM predicates? i can't grok "Artifacts Evaluated - Functional" and i can't find info on-line

Please see the below descriptions.

I would like to ask you to use this only internally for FSE-internal purposes,
because this is not yet published.

Best,
Dirk

We recommend that three separate brands related to artifact review be associated with research
articles in ACM publications: Artifacts Evaluated, Artifacts Available and Results Validated.
These brands are considered independent and any one, two or all three can be applied to any
given paper depending on review procedures developed by the journal or conference.

Artifacts Evaluated
This brand is applied to papers whose associated artifacts have successfully completed an
independent audit. Artifacts need not be made publicly available to be considered for this brand.
However, they do need to be made available to reviewers. Two levels are distinguished, only one
of which should be applied in any instance:
• Artifacts Evaluated – Functional
The artifacts associated with the research are found to be documented, complete,
exercisable, and include appropriate evidence of verification and validation.
Notes
Documented: At minimum, an inventory of artifacts is included, and sufficient
description provided to enable the artifacts to be exercised.Consistent: The artifacts are relevant to the associated paper, and contribute in some
inherent way to the generation of its main results.
Complete: To the extent possible, all components relevant to the paper in question are
included. (Proprietary artifacts need not be included. If they are required to exercise
the package then this should be documented, along with instructions on how to obtain
them. Proxies for proprietary data should be included so as to demonstrate the
analysis.)
Exercisable: Included scripts and/or software used to generate the results in the
associated paper can be successfully executed, and included data can be accessed and
appropriately manipulated.
• Artifacts Evaluated – Reusable
The artifacts associated with the paper are of a quality that significantly exceeds minimal
functionality. They are very carefully documented and well-structured to the extent that
reuse and repurposing is facilitated. In particular, norms and standards of the research
community for artifacts of this type are strictly adhered to.
Artifacts Available
This brand is applied to papers in which associated artifacts have been made permanently
available for retrieval.
• Artifacts Available
Author-created artifacts relevant to this paper have been placed on a publically accessible
archival repository. A DOI or link to this repository along with a unique identifier for the
object is provided.
Notes
We do not mandate the use of specific repositories. Publisher repositories,
institutional repositories, or open commercial repositories (e.g., figshare or Dryad) are
acceptable. In all cases, repositories used to archive data should have a declared plan
to enable permanent accessibility. Personal web pages are not acceptable for this
purpose.
Artifacts do not need to have been formally evaluated in order for an article to receive
this brand. In addition, they need not be complete in the sense described above. They
simply need to be relevant to the study and add value beyond the text in the article.
Such artifacts could be something as simple as the data from which the figures are
drawn, or as complex as a complete software system under study.
Results Validated
This brand is applied to papers in which the main results of the paper have been successfully
obtained by a person or team other than the author. Two levels are distinguished:
• Results Replicated
The main results of the paper have been obtained in a subsequent study by a person or
team other than the authors, using, in part, artifacts provided by the author.•
Results Reproduced
The main results of the paper have been independently obtained in a subsequent study by
a person or team other than the authors, without the use of author-supplied artifacts.
In each cases, exact replication or reproduction of results is not required, or even expected.
Instead, the results must be in agreement to within a tolerance deemed acceptable for
experiments of the given type. In particular, differences in the results should not change the main
claims made in the paper.