IPEP 14: Notebook Regression Tests

Zachary Sailer edited this page Aug 3, 2013 · 6 revisions
Clone this wiki locally
Status Active
Author @dwyde
Created April 2013
Updated April 2013


The HTML notebook currently lacks unit or integration tests. Regression tests would help with potential refactoring, and are generally good to have.

Features to test:

  • User interface: toolbars, keyboard shortcuts
  • JavaScript code
  • Code cell execution
  • Code output event handling / rendering
  • Text cell (Markdown / LaTeX) rendering

Test Frameworks


CasperJS is a JavaScript test utility built on top of PhantomJS, a headless WebKit implementation.


  • Can test code cell execution from JavaScript
  • Multi-page navigation
  • xUnit-friendly output option


  • Extra software dependencies


Selenium is a web browser automation library with Python bindings. It can be installed with pip.


  • Integrates with Python's standard unittest module
  • Can test code execution in the kernel


  • Tests still need to execute strings of JavaScript, for example, to get text from CodeMirror input areas.
  • Relatively slow
  • An extra software dependency

Prep work

Customize whether to run tests in Firefox, Chrome, Internet Explorer, etc.

  • Use a wrapper script, or a config file

Make the tests aware of a notebook server's URL

  • Run a notebook server in a subprocess, and write the port to the test process? This is straightforward with multiprocessing.


QUnit is a JavaScript unit test framework.


  • No additional software dependencies: just HTML, JavaScript, and CSS.


  • Can't easily test executing code in a kernel


  • Can it directly test the menubar and other UI elements, or just JavaScript models?

Prep work

Create an HTML page that includes the notebook's JavaScript, and QUnit's code.

  • Inherit a tests page from the notebook.html template, and add an associated handler to the Tornado app


Jasmine is a JavaScript behavior-driven development (BDD) framework. It's a head-to-head competitor with QUnit, modeled after test tools in Ruby on Rails.


Behavior-driven development, if you're a fan.


DOM-less test runner?

Prep work

Similar to QUnit.


The notebook's regression tests should check that code cell execution works, which is a vote for either CasperJS or Selenium.

CasperJS has less of an impedance mismatch for testing JavaScript - Selenium tests have to write JavaScript code as strings.

Related discussions