IPEP 14: Notebook Regression Tests
Clone this wiki locally
The HTML notebook currently lacks unit or integration tests. Regression tests would help with potential refactoring, and are generally good to have.
Features to test:
- User interface: toolbars, keyboard shortcuts
- Code cell execution
- Code output event handling / rendering
- Text cell (Markdown / LaTeX) rendering
- Multi-page navigation
- xUnit-friendly output option
- Extra software dependencies
Selenium is a web browser automation library with Python bindings. It can be installed with
- Integrates with Python's standard
- Can test code execution in the kernel
- Relatively slow
- An extra software dependency
Customize whether to run tests in Firefox, Chrome, Internet Explorer, etc.
- Use a wrapper script, or a config file
Make the tests aware of a notebook server's URL
- Run a notebook server in a subprocess, and write the port to the test process? This is straightforward with
- Can't easily test executing code in a kernel
- Inherit a tests page from the notebook.html template, and add an associated handler to the Tornado app
Behavior-driven development, if you're a fan.
DOM-less test runner?
Similar to QUnit.
The notebook's regression tests should check that code cell execution works, which is a vote for either CasperJS or Selenium.