Skip to content

Latest commit

 

History

History
73 lines (54 loc) · 5.41 KB

ListOfArtifacts.md

File metadata and controls

73 lines (54 loc) · 5.41 KB
Raw

List of known research artifact types

For a discussion on why we might want to divide our work in to the following research artifacts, see the SE research reuse manifesto.

IMPORTANT NOTE: This list is not exhaustive. In fact, we store it here on Github so that the broader community can refine this document. So if your artifact is not included in the above, then write an issue in this repo to propose a new artifact type:

Preamble

"Artifacts": Why?

Researchers write papers but research should be more than about papers. Rather, workshops, conferences and journals should be a place to visit to find “things” that are useful for a wide range of tasks.

To these end, we ask researchers to “take some scissors” to their work and divide them into the separate “chunks”. These chunks, which we will call “artifacts” are the support materials that enabled the creation of results published in a paper. These artifacts are reusable units of research that can be used to support many other research endeavors.

"Artifact": What are they?

Some artifacts are executable (e.g. scripts) and some are not (e.g. tutorials).

The following list is sorted by artifact size, from smaller to bigger. About two-thirds the way through the list we arrive at "the actual text of an author's paper" so:

  • All the itemes before that point are generally smaller in size than a standard research paper (i.e. 10pt, two column, 10 pages).
  • All the items after that point are bigger than a research (e.g. a paper might use some scripts or data)

Warning: "Artifacts" need Justification

An author of an artifact has to offer some commentary on why a particular artifact is worthy of consideration. We recommend a short (say, one page) commentary explaining why anyone else should care about the artifact. That commentary should include

  • Synopsis, authors' contact details.
  • Optinally, consider applying the standard “Design Patterns” Coplien template to explain the artifact (name, context, forces, solution, resulting context, rationale; for examples, see https://goo.gl/WJFmX1)
  • For large artifacts, add the download site.
  • For smaller paper-based artifacts, attached the artifact itself to the end of the justification.

Note that the shorter the artifact, the great the onus on the author for demonstrating that the artifact is worthy.

  • For example for ant-patterns, there needs to be some support text and/or results arguing that it is an anti-pattern actually seen in practice and that the anti-pattern is a significant problem.

List of artifacts

  1. Motivational statements or reports or challenge statements or lists of open issues that prompt an analysis;
  2. Hypotheses, about expected effects in some area;
  3. Checklists used to design the analysis (see also, the Checklist Manifesto;
  4. Bibliographies, comprehensive, annotated, and insightful (e.g. showing the development or open areas in a field);
  5. Study instruments such as surveys interview scripts, etc;
  6. Statistical tests used to analyze results (along with some notes explaining why or when this test is necessary);
  7. Commentary on scripts used in the analysis;
  8. Examples of particularly informative visualizations (e.g. Sparklines http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR )
  9. Baseline results against which new work can be compared;
  10. Sampling procedures e.g. ``how did you choose the projects you studied?'';
  11. Patterns describing best practices for performing this kind of analysis;
  12. Anti-patterns describing cautionary tales of ``gotchas'' to avoid when doing this kind of work;
  13. Negative results that are anti-patterns, backed up by empirical results;
  14. Tutorial materials: Guides to help newcomers become proficient in the area. Some of these tutorial materials may be generated by the researcher and others may be collected from other sources.
  15. New results that offer guidance on how to best handle future problems.
  16. Future work: From the results, there many be speculations about open issues of future issues that might become the motivation for the next round of research.
  17. The actual text of an author's papers;
  18. Any data used in an analysis
    • Either raw from a project;
    • Or some derived product. Note that some data is too large to fit into the standard on-line freely available repos (e.g. Github only allows 1GB reps). For such data, we suggest using some file XXX.goto; each line of which is one url where the related data can be collected.
  19. Scripts used to perform the analysis (the main analysis or the subsequent statistical tests or visualizations; e.g. the Python Sparklines generator). Scripts can also implement some of the patterns identified by the paper.
  20. Executable models that can generate exemplar data; or which offer an executable form of current hypotheses;
  21. Delivery tools to let novices automatically rerun the analysis; e.g.
    • Config management files that can
      • build the system/ paper from raw material and/or
      • update the relevant files using some package manager
    • Virtual machines containing all the above scripts, data, etc, pre-configured such that a newcomer can automatically run the old analysis.