Time-travel debugging interface

[nwinter]

The biggest problem when trying to do harder levels in CodeCombat is that players have only one way of debugging their code:

this.say("Rect is at " + rect.x + ", " + rect.y + " with size " + rect.width + " x " rect.height");

It took me 30 seconds just to type that out, and I made a syntax error. This is not okay. As a Hacker News commenter put it:

"Only the very darkest realm would deprive its wizards of the mighty printf spell! To what Silicon demon legion did you pledge fealty to summon this coding abomination upon an unsuspecting land? Winter is truly coming at last."

So why is debugging so bad? Well, for one, serializing things across the web worker boundary is imperfect--you can't pass most World objects because they'll point to Thangs and Vectors and such which contain functions--and very slow if you're sending strings for each step of your algorithm for each frame of the simulation. So not only do you get really boring and unhelpful log statements like "Rect is at [Object object]", but it's very easy to blow up the simulation performance, and you have to either add a better UI than this.say() for seeing the messages or force the player to go to the browser console. We used to do this, but we shut it off.

We could solve most of these problems, but there is a better way. An amazing way. Just lurking, waiting for us to have time to implement it. CodeCombat's programming environment should allow us to build a time-travel debugging interface that combines the best of both breakpoint-based stepping debuggers and using log statements, but far better than either. It'll be quite Victorian. You'll just scrub back and forward through your code's execution history and hover over any variable to see its value at any point in time.

jsdares actually does something like this already:

In its solution, each statement generates a step message, and when you over over that statement's frame number, it shows the step message below the statement. It also uses blue dots to show which statements were recently executed or will execute. Finally, on the left you can see it shows the values of all currently defined variables. @janpaul123's thesis discusses the interface and motivations in Chapter 3, although the implementation has advanced beyond the thesis since its writing. There is some discussion of how it's done in Chapter 4.

I was thinking of implementing something different. Instead of trying to show all variables and their values, I'd prefer to focus on one variable when the player hovers over it. CodeCombat variables are often complex objects like Thangs, so a simple string isn't going to be enough to get a good picture of their state. You might want to see a Thang's pos property, or perhaps even its target.target.id to see who its enemy is attacking. Clearly we're not going to serialize all that state for every variable for every frame for every Programmable method.

Instead, what I plan to do is to keep the simulated World around in the web worker thread, possibly more than one at various intervals to make things even faster, and resimulate up until the current call whenever this kind of debugging info is requested. Then we can use Aether to thoroughly inspect the state of all accessible properties, serialize everything we can, and send it over for custom presentation that's aware of our common types (like Thangs and Vectors).

I think the delay should be just a split second on most levels--suitable for idly hovering over various variables in the code to check them out. Then when the player wants to see a variable change over time, she can just scrub the playback and watch the variable change. It should be especially fast scrubbing forward, since we'll already have the World simulated up until the current frame and can just get the next frame--but it shouldn't be too bad going backwards, either.

With some cleverness plus the insanity of Aether's control flow yielding, we should also be able to scrub/step forward and backward within the current function execution on a statement-by-statement basis. We probably can't use the main playback scrubber for this, but perhaps some keyboard shortcut or traditional step forward/back/over/out buttons for when one wants to get really fine-grained could do the trick. jsdares experimented with a second timeline sort of interface, since beginners are confused by stepping debugger buttons, but as I recall having two was also confusing.

To implement this, we'll also need to do a bunch of improvements over in the Aether project, where I've added a companion issue. This issue can track the progress on the front-end interface as well as the coordination of Aether state harvesting and World resimulation in the background thread.

If we can pull this off well (and polish up some other editor rough edges), it has the potential to turn CodeCombat into the best environment for solving programming challenges that the world has ever seen, since not only will be there a sweet visual interface to see what's going on and gameplay mechanics to make it fun, but the overly powerful debugging, live-coding, and co-op multiplayer should make it possible to solve harder algorithms than you'd be able to do in a traditional environment--programming without a blindfold on, as it were.

[janpaul123]

Just a warning: jsdares' implementation is rather comprehensive, in fact, I think I spent most of the time working on jsdares just to make this work. But the most advanced features (time scrubbing, code scrubbing) are actually not used by most people. This might be because for the simpler puzzles they are not really necessary, but also because they might be too hard to use.

I still believe that just having all kinds of data (yay, Aether!) is great for trying out different visualisations. Inspecting specific variables might be a great one to start with. A visualisation that I think works well for jsdares is visualising method calls by highlighting corresponding elements on the canvas. Just be careful -- this may be a great time-sink as chances are that you won't get the great debugger you're hoping for on first try. But it might well be worth it, too. :-)

[nwinter]

A challenge! Yeah, I wasn't prioritizing this until we did Gridmancer–the need for it in harder challenges became painfully clear then.

[nwinter]

Made some progress on this–see screenshot at end of the Gridmancer blog post.

[janpaul123]

Sweet, looks good! Keep us posted on any user testing or other results about this, I'm very curious to hear if and how learners use these controls. :)

[And, in any case, keep up the great work with CodeCombat, awesome to see your open approach]

[nwinter]

@janpaul123 I haven't really finished it, it's not too pretty, it's often slow, there are no step messages, there's no time-abstracted variable display, and there are many edge cases left to handle for which variables can be inspected, but you can kind of get the idea now:

Would love any feedback.

[nwinter]

Some remaining UI tasks:

• Do something where it's not just a <ode> tag with typography: "terrible"
• Add timelessness to SpellDebugView for simple values (think a sparkline for numbers)
• Give SpellDebugView a nicer interface for displaying Thangs (Portraits? full ThangAvatarViews?)

[janpaul123]

Looks great. I guess we all know from using debugging tools that this is very valuable. This solves a real problem reported by users, so that's good!

Just want to reiterate what I said earlier: would love to hear about user testing results! The scrubbable timeline didn't work too well in jsdares (see also codecombat/aether#10), so it's critical to test how learners actually use this. Might also be worthwhile to just prototype/hack different visualisations (this one, step messages, breakpoints, jumping to function calls when clicking a function, showing the flow through a program a la Learnable Programming, etc) and test if they work before fully implementing them.

[nwinter]

I agree that it's all about UX testing it. I'm guessing we'll have some different results, because we implemented this for more advanced challenges with more experienced programmers rather than the beginners, since we saw (and you mentioned) that beginners don't take advantage of the debugging tools.

[janpaul123]

👍

[AwokeKnowing]

@nwinter The debugging hover tool is very useful for me. But I also add a lot of variables to "this", and when I hover over them, it says "this.myvar is undefined". Naturally, the variables I add are the ones most in need of debugging, so it would be very very helpful if the hover could show the variables that have been added to "this".

[nwinter]

@schmatz this relates to what you've been working on recently.

[schmatz]

@nwinter Thanks for tagging these things for me, I'll keep an eye on them as I work on this.

[jpritikin]

Any progress on the debugger? This feature would make a huge difference in the ability of students to self study.

[nwinter]

We just shipped the interpreter, so now it should be a lot more performant to inspect variable state than the hacky method we were using before, and it'll actually do scoping correctly and all that stuff. We will still need to improve the UX a lot, though, before it will be helpful to beginners.

The current state of the code: it's turned off here, and it now needs to be updated to work with the way that the interpreter reads variable state. @basicer can advise on what needs to be done there. Then we'll have to fix whatever rust it's gathered from being turned off for so long and design a new, simpler user interface for it.