Rosetta design doc

This page documents the initial design of the Rosetta runtime code visualization toolkit, based on experiences and notes collected over the past few years.

Created on 2015-12-16 by Philip Guo

Overview

Rosetta aims to be an expressive toolkit for creating runtime code visualizations like the ones that power Python Tutor. Right now the Python Tutor visual primitives and layout algorithms are hard-coded in JavaScript with a hacky mix of d3, jQuery, and jsPlumb. I wrote the current viz code back in 2011, with some tweaks in 2012.

There is only one canonical visualization format (with a few select variants for power users), which works fine in practice but is nearly impossible to customize for different pedagogical use cases. Right now it's "here's your visualization, if you don't like it, too bad!" I've managed to bolt on support for other languages such as Java, JavaScript, TypeScript, Ruby, C, and C++, but the viz code gets increasingly hackier with each new language. This "ossification" in the current code base precludes future innovations and research contributions.

Thus, I want to take what I've learned from the past 6 years of Python Tutor usage and leverage that experience to create a more expressive toolkit for exploring future runtime code visualization ideas. The ultimate dream is to be able to use Rosetta to easily build any kind of runtime code visualization for any language for the purposes of both learning and production-scale debugging. A more realistic dream is to simply target general-purpose imperative and OO languages, and to focus on the learning use case; in other words, to replicate what is already possible on Python Tutor, with a substantial cleanup of the underlying viz code base to make it more general-purpose and extensible in the future.

One major inspiration for Rosetta is Vega and related toolkits by Jeff Heer's group at UW.

Primary Design Goals

• Multi-language: must be expressive enough to create visualizations in many general-purpose imperative and OO languages such as Python, Java, JavaScript, Ruby, C, and C++
• Custom layout specifications: must be able to specify object layout rules in some simple yet expressive format (without resorting to writing full-blown custom JavaScript code like I currently do for Python Tutor)
• Visual transitions and diffs: show what has changed or will change between two consecutive program points, either via simple fade in/out, colored outlines around relevant objects, or unified diff (diff -u) sorts of views. This feature is crucial for helping users grok state changes, although it doesn't need to mean fancy eye-candy animations. In fact, simpler and more static is better, since those can be printed out in a static rendering (e.g., a PDF textbook).
• Persistent show/hide: the user should be able to show/hide elements of the visualization and have those choices persist across program points as they step through the visualization. i.e., if the user hides/collapses an object X and then steps forward, that object should still be hidden/collapsed.
• Ubiquitous annotations: all visualization elements must support inline textual annotations, so that users can write explanations of what is going on as the target program executes (e.g., to create a tutorial).
• Polymorphic views: a built-in object in the target program must be able to be visualized in multiple ways to support different levels of abstraction. For instance, a C string may be viewed as either a string literal, an array of chars (i.e., letters), or an array of bytes (in decimal, octal, hex, etc.). A C/C++ union may be viewed in multiple ways. A Python/JS/Ruby/etc. object instance may be shown either as its constituent fields or pretty-printed using its toString-like method.
• Custom data structure views: Make it possible to write custom renderers for more sophisticated objects beyond basic data structures. For instance, a specially-made Graph object may be rendered as a d3-inspired node-and-edge graph. A pair of numerical arrays may be rendered as a scatterplot. A 2-D matrix of RGB values may be rendered as a bitmap image (e.g., for Media Computation).

Secondary Design Goals

These features are nice-to-have but not mandatory:

• Scalability: for production-scale debugging, it would be nice to scale up to larger data that cannot fit on a whiteboard-like surface in the real world. e.g., compressing a 10000-element list into, say, a line chart, then allowing the user to semantically zoom. also selectively showing and hiding objects on-demand so as not to overwhelm the user.
• Animations: showing how objects mutate by animating motion is a nice gimmick, but my hunch is that it doesn't add much beyond simple visual transitions, which is a primary design goal
• Interactivity: Classic research has shown that actively playing with a code visualization is much more effective for learning than simply passively viewing it. Support interactivity such as blanking out object values and making learners guess what value goes where, making learners drag and drop pointers to the right places, or other sorts of "micro-quizzes" to get learners more engaged. This goal is secondary since the primary goal of Rosetta is to generate visualizations, not rich learning activities. But it's got a lot of potential for future work!
• Functional and dataflow programming: it would be cool to also visualize functional or dataflow languages in addition to the state-based visualizations optimized for imperative/OO languages
• Semantic zooming: similar to polymorphic views, but support different levels of granularity in the visualization, controlled via zooming
• Mobile/touch friendly: can explore gesture-based interactions in the future
• Freeform drag-and-drop: allow the user to freely drag and drop any element in the visualization, and have those positioning decisions persist across program points.
• Holistic time-expanded visualizations: Rosetta is designed to show static snapshots of runtime state, optionally with diffs between consecutive states. It helps learners form imperative mental models. It's a bigger stretch to extend it to show "holistic" views of all states at once, expanding time into space, like in some of Bret Victor's Learnable Programming demos.

Note that I want to avoid feature creep; if I tried to expand Rosetta to handle all the secondary cases, the project would bloat up to the point where quality and cohesion suffers, and also be so watered-down that it might just devolve into programming in raw JavaScript/d3/etc. again. Beware the Turing Tarpit.

Implementation ideas

Possible building blocks: React, jQuery, jsPlumb (for arrows)

Not sure if d3 is the right level of abstraction here, though. However, ideally I'd want something that can animate state changes so that the user can see what has changed between two consecutive states. d3 does this well, but does React do this?

I like Vega's idea of perhaps making Rosetta into a declarative "file format" for visualizations in, say, JSON or maybe YAML format. That way, a human can easily generate a new kind of visualization without writing a bunch of JavaScript code. (TODO: maybe JSX from Facebook is also good since it lets you mix JS and markup?)