Adding record-and-playback to IPython Notebook for teaching #2832

Open
gvwilson opened this Issue Jan 22, 2013 · 24 comments

Comments

Projects
None yet
9 participants

Software Carpentry is moving its Python lessons (and eventually other content as well) over to the IPython Notebook. As part of that, we would like to replace our video screencasts with something that replays a Notebook session in the browser, in sync with an audio voiceover. We want to do this because:

  1. People can't pause a video, then copy and paste its text, because there is no text: there are just pixels arranged to look like letters.
  2. For the same reason, the code in videos is invisible to search engines.
  3. We also can't diff and merge videos in version control the way we can diff and merge source code and Notebooks.
  4. And video is inherently less adaptable to new platforms (e.g., tablets and smart phones) than first-class content that's rendered in the best way for whatever device the user has.

In the long run, we'd like users to be able to:

  1. Press "Play".
  2. Watch the Notebook reconstruct itself in sync with a voiceover.
  3. Press "Pause".
  4. Do some experimenting (e.g., fill in a short exercise, play with parameters for graphs, etc.).
  5. Press "Reset" to put things back the way they were.
  6. Press "Play" again to continue with the "main line" of the presentation.

Implementing this kind of "what if?" will be difficult, because we'd have to clone and restore the state of the back-end engine, but a "replay only" tool would still be very useful: at least as useful as our existing screencasts. From discussion with Brian Granger, Min RK, and Bobby Richter (of Mozilla), it looks like "replay only" could be done as a plugin without any changes to the Notebook itself:

  1. Popcorn.js and a little bit of custom Javascript can be used to synchronize a soundtrack (e.g., an MP3 voiceover) with updates to the Notebook's cells.
  2. The Notebook now allows per-cell metadata, so timing information (e.g., when to start displaying this cell and how long to take filling it in) can be embedded without format changes.

The workflow we envision is:

  1. Author spends minutes, hours, or days incrementally building up the notebook.
  2. Author puts notebook in "mark time" mode.
  3. Author records audio her favorite desktop tool while clicking "next" in the notebook to insert time marks into metadata.
  4. Author saves audio and inserts URL into notebook.
  5. Time passes.
  6. Audience opens notebook in browser (either local or web copy---audio file may be local or web as well).
  7. Audience presses "Play".
  8. Notebook incrementally reconstructs itself in sync with voiceover.
  9. Audience can use "Rewind", "Fast Forward", "Pause", "Go To Cell" (jumps the audio) or "Go To Time" (jumps the notebook) at will.

Please add comments on this ticket if you have suggestions or design ideas, or want to help build, test, and use this widget.

Owner

Carreau commented Jan 22, 2013

You will probably need cell-id for it to be easier.
The other question is do you want live code "highlighting"
it seem to be possible with the new version of code mirror.

I'll see if we can hook on codemirror to record the typing.

I think already having a per-cell replay with info would be good,
potentially using the new cell toolbar.

Contributor

ptone commented Jan 22, 2013

One of the issues with a "rewind" in a simplistic implementation - is that if new cells are appended - they will be appended a second time in the second run through.

I had put my rather quick stab at this here: https://github.com/ptone/nbcast

@yihui yihui referenced this issue in yihui/knitr Jan 22, 2013

Closed

Popcorn.js #466

jasonm commented Jan 24, 2013

I just want to say that I love this idea a bunch 😃

Owner

ellisonbg commented Jan 29, 2013

Here are some of the ideas we talked about for getting a quick and easy initial version of this working.

  • We will encode the cell level playback information in cell level metadata. It will have things like: should the cell fake type itself out, how long to wait before showing the output, how long to wait before showing the next cell, etc.
  • Let's start out with a very primitive "record" mode that consists of manually editing the cell metadata. This is only to help us understand what that metadata needs to be.
  • The initial playback mode should be a simple plugin to the notebook that puts a new "playback" button in the toolbar, hides all the cells and then looks at their metadata to decide how to play them back.
  • Once we have iterated on the metadata format and playback mode, then we can think about the record mode.

Cheers,

Brian

Owner

ellisonbg commented Jan 29, 2013

I should also note that I think we won't need to use popcorn.js to synch the audio and notebook playback. That would be overkill for such a simple thing.

Contributor

wking commented Jan 29, 2013

On Mon, Jan 28, 2013 at 08:09:32PM -0800, Brian E. Granger wrote:

  • We will encode the cell level playback information in cell level
    metadata. It will have things like: should the cell fake type
    itself out, how long to wait before showing the output, how long
    to wait before showing the next cell, etc.

Does this mean that all the recorded cells will be stored in the
notebook file, except that some (most) will be hidden on the initial
load? Hidden cells would have a “created” timestamp, and all recorded
cells could have a “destroyed” timestamp. If a single cell is filled
in with text, rendered, edited, and rerendered, how would that be
recorded?

It seems like it would be easier to do this at the keystroke level
with a cursor abstraction, but maybe that makes rewinding too
difficult. On the other hand, there's no rewinding in the nbcast
demo, so maybe play+reset is enough functionality.

On 2013-01-29 9:41 AM, W. Trevor King wrote:

Does this mean that all the recorded cells will be stored in the
notebook file, except that some (most) will be hidden on the initial
load? Hidden cells would have a “created” timestamp, and all recorded
cells could have a “destroyed” timestamp. If a single cell is filled
in with text, rendered, edited, and rerendered, how would that be
recorded?
It wouldn't be in version one: the final state of the cell would replay
itself at a uniform speed. Simplest thing that could possibly work...
We'll worry about typeahead/delete replay later, if/when there's clear need.

On 2013-01-28 11:11 PM, Brian E. Granger wrote:

I should also note that I think we won't need to use popcorn.js to
synch the audio and notebook playback. That would be overkill for such
a simple thing.

What would we use instead?

Contributor

wking commented Feb 6, 2013

I've tweaked @ptone's nbcast to move cueing data into the IPyNB JSON
1. How should this be polished up, because it will clearly need a
lot of polishing before it's ready for a merge into IPython. @ptone,
do you want PRs against ptone/nbcast?

Contributor

ptone commented Feb 9, 2013

pull requests probably don't make sense, my work was mostly intended as proof-of-concept, not a starting point for merge per se. This issue should be the canonical place to track this idea - so linking your fork seems the right way to do it.

It seems a fundamental design decision is whether the media and its timeline should live at the notebook level and drive the cell playback through something like popcorn, or should some interface inside the cells trigger the relevant part of the media to play. That is, do cells cue and play the media based on their metadata, or does something set up popcorn cues based on cell meta data. The nbcast project adopts more of the popcorn oriented approach, as it seems easier to flow a sequence of cells that way, but having the playback driven by the cells makes it easier to jump around arbitrarily between cells.

Contributor

wking commented Feb 9, 2013

On Fri, Feb 08, 2013 at 07:46:14PM -0800, Preston Holmes wrote:

That is, do cells cue and play the media based on their metadata, or
does something set up popcorn cues based on cell meta data.

I'm not entirely clear on the distinction here. It sounds like the
former is “when a cell executes, trigger other event(s)”. This could
also include triggering other cells, in order to make a staged reveal.

It sounds like the latter means doing what nbcast already does and
just attaching cell reveals to a single, master timeline.

The nbcast project adopts more of the popcorn oriented
approach, as it seems easier to flow a sequence of cells that way,

Agreed.

but having the playback driven by the cells makes it easier to jump
around arbitrarily between cells.

Not agreed ;). Having a master timeline means you don't have to worry
about the case where some cell was executed out of it's usual order,
and triggered an additional cue cascade. I don't know enough popcorn
yet to hide my cells on rewind/reset, but the action you have to take
is obvious. If a cell is only displayed within a given time range,
the visibility is only a function of the current time (i.e. it is
independent of timeline manipulation leading up to the current time).

Contributor

wking commented Mar 1, 2013

I've fixed up my version of @ptone's nbcast to avoid calling sort_cells() (removed by b2e230d). The current version of nbcast (wking/nbcast@803b357) works with the current IPython tip (8051036) as well as versions back through at least 50a5838.

@wking wking referenced this issue in swcarpentry/DEPRECATED-website Mar 3, 2013

Closed

Build Popcorn-based replay-with-audio plugin for IPython Notebook #15

lrq3000 commented Mar 21, 2013

Just to say that this would be genius. IPython Notebook has a great potential for training-based learning, similar to what Coursera offers, but a lot more easier to setup.

Contributor

wolever commented Apr 17, 2013

For those interested, I've started work on this plugin, here: https://github.com/wolever/browsercast

I've also got a short screencast showing the basic functionality, here: http://youtu.be/CYxZq4rZw0c

Still very proof of concept, so feedback would be great!

Contributor

wolever commented May 22, 2013

BrowserCast is now now a non-proof-of-concept useful thing! The source is here: https://github.com/wolever/browsercast

Here are a couple old-school screencasts which show it off:

And you can try it for yourself with one bookmarklet: http://wolever.github.io/browsercast/

Contributor

ahmadia commented May 22, 2013

Bravo!

That is quite impressive. I'll definitely be taking a deeper look into
this.

On Wed, May 22, 2013 at 10:18 PM, David Wolever notifications@github.comwrote:

BrowserCast is now now a non-proof-of-concept useful thing! The source is
here: https://github.com/wolever/browsercast

Here are a couple old-school screencasts which show it off:

And you can try it for yourself with one bookmarklet:
http://wolever.github.io/browsercast/


Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/2832#issuecomment-18308662
.

Contributor

ptone commented May 24, 2013

Looking really nice @wolever - what do you think the feasibility is of eventually doing the audio recording right in the browser someday - bleeding edge for sure.

http://www.html5rocks.com/en/tutorials/getusermedia/intro/

Contributor

wolever commented May 24, 2013

Thanks! I actually spent a bit of time talking with some people at Mozilla Toronto about audio recording, and right now I don't think it will be worth the effort. My current understanding is that the getUserMedia() APIs are still somewhat in flux, and don't necessarily work that well (just for ex, the recording demo on the site you linked returns a blank audio file with the most recent stable Chrome, and doesn't work at all with Firefox)… In a few months once those settle down, though, it might well be worth another look.

Contributor

wolever commented May 24, 2013

I've created an issue for in-browser audio recording: wolever/browsercast#6 and I'll post updates there.

Presume this is stale?

Owner

Carreau commented Jan 12, 2016

We can migrate to the new repo. With the current refactoring in progress, this will be much easier to do in 6month or so. We have a meeting this Friday to discuss about how to store/transmit application state, so I'll mention that explicitly.

lrq3000 commented Jan 12, 2016

Great news! This feature would be amazing and I know for sure I would use it extensively! 💯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment