Study design

[kirangadhave]

The minimum evaluation we want is user feedback on the usefulness of our techniques/plugins.
Good to have - Quantitative study

Qualitative

Participants: ~5
Discuss potential candidates with Klaus.
Dr. Kogan has a class with students using notebooks (will they have their data?)

• IRB
• Brief interview about their existing notebook workflow
• Discussion on pain points
• Brief introduction to the technique
• A short demo (maybe a video?)
• Participants load one of their notebooks in Jupyter with IDE installed.
  OR
• Start a new analysis with their data.
  OR
• Task with an analysis to do with our data
• (Re)analyze with trrack support + interactive visualizations.
• Feedback interview
  • experience (learning curve, difficulty, etc.)?
  • were any pain points addressed
  • what did you like/dislike

#34 - for tasks

Quantitative

Participants: not sure
Crowdsourced on prolific
Dr. Kogan's class (might be able to do in-person study)

• IRB
• Demo Video
• Tasks
  • Perform analysis in two notebooks: regular jupyterlab & IDE-enhanced jupyterlab
  • Similar tasks with different dataset
  • Goal is to get time to completion and accuracy
• Feedback Questionnaire

[kirangadhave]

A good framework is to look at (A) What's Wrong with Computational Notebooks? paper and build the evaluation tasks around common problems that we can address.

e.g

• Setup. Trrack helps us replay data-cleaning steps
• Explore and analyze. Auto-apply interactions on changes to upstream code
• Reproduce & Reuse. Can reproduce the interactive analysis parts of the notebook. (Maybe reuse?)
• Share and collaborate. Interactive analysis parts can be tracked and history preserved.

Also for supplementary material?