Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Selenium web tests proof-of-concept #1493

Closed
wants to merge 4 commits into from

5 participants

@stefanv

This PR provides a starting point for adding web tests to IPython.

Use easy_install or pip to install the selenium package, and then run iptest-web. Currently, the scripted browser is Firefox.

I made use of the PageObject pattern--more here: http://pragprog.com/magazines/2010-08/page-objects-in-python

It will be interesting to see how the web tests eventually influence page design. E.g., at the moment I only have one test which creates a new notebook from the dashboard and then renames it. In order to script that test I had to jump through a couple of hoops, since many of the elements, such as buttons, are not uniquely identified by IDs.

@ellisonbg
Owner

I looked a bit more at Selenium and I am not sure it is going to be the best way of testing our stuff. Honestly, I think it would be a much better match to be able to write tests in Javascript/jquery. We should look into such alternatives before merging this.

@stefanv

Testing the javascript itself is easy--but making the browser click on buttons etc. is harder. If you can suggest a library that will allow us to do that, I'd gladly investigate.

@ellisonbg
Owner
@stefanv

I don't think one can construct even the simple test case I have here in QUnit:

  • Load the dashboard
  • Create a new notebook
  • Rename the notebook

JUnit is fine for testing the javascript library part of things, but for the rest you'll probably need something like

http://www.testjquery.com/

@stefanv

Another free alternative is http://www.getwindmill.com/

@fperez
Owner

I'm all for writing as many tests as we can cleanly at the unit level in JS, but it seems to me that having some of these integration tests from the browser side would also be immensely valuable.

Basically with the notebook we're in a very similar position now to where we were a few years ago in the terminal IPython, when we started adding tests to the main code for the first time. At the beginning we just added a few high-level tests that simulated interactive usage, and while a but coarse and crude, they were already very useful in 'anchoring' our stuff somewhere. Over time we've become much, much better at increasing our test coverage at a very fine level, and now I actually feel reasonably OK with our overall test suite (I know it's not perfect, but it's far from being bad).

So I see this in a similar light: it provides a few big tests of the main user-facing experience, that at the very least will help us not break things in gross, catastrophic ways due to some small omission. I remember with fright how early on, the only way for me to know something hadn't broken in IPython was to try a bunch of things interactively and hope I was exercising it enough. That's basically where we are with the notebook right now, and I'm again quite afraid.

I think some of these coarse tests will be super useful with the upcoming planned JS refactorings: they will serve as a safety net for the main stuff, and can help us in making the refactoring cleaner.

Obviously they don't preclude that we start writing proper, fine-grained tests in pure JS, and I'm all for that. But I think the two kinds of testing are complementary, and that we really should consider this, or something like this, very seriously.

@ellisonbg
Owner

I don't disagree one bit that we need tests. I am just not at all sold on Selenium. The idea of writing browser focused tests in Python just doesn't make sense to me. When you look at the Selenium tests, it is clear that they have basically rewritten jQuerry in Python for talking to web browsers. We have to learn yet another DOM API and you can see from the tests that there is a lot of impedance mismatching that you have to do.

@stefanv

We may need both for complete coverage: Selenium-style tests for checking integration, and Javascript tests for exercising a single test notebook. Whenever you leave a page, Javascript testing becomes a pain (you have to let the browser know somehow that it has to execute code on the newly loaded page), but there's still a lot that can be done without it. I'll have a look at integrating QUnit when I'm back on Wednesday.

@ellisonbg
Owner
@fperez
Owner
@ellisonbg
Owner
@fperez
Owner
@fperez
Owner

Just as an FYI, I just saw this which might be worth taking a look at: http://jeanphix.me/Ghost.py. It's Qt based, but it lets you programatically control a webkit instance.

@fperez
Owner

I talked with @ctb a fair bit about this question; Titus literally wrote the book on the matter, so I'm inclined to give his advice some serious consideration. He also pointed out that if we go with selenium, we can use Sauce Labs for CI for the web tests, including multiple browser support.

@ellisonbg, could you try to clarify precisely what problems you foresee with using Selenium? Perhaps @ctb can help us out given his experience, and we can make a decision.

@ctb
ctb commented
@ellisonbg
Owner

Imagine if I told you that there was this fantastic new testing framework for Python. Imagine that this new library allows you to write tests for your Python code using Ruby by calling into the Python/C API and translating Python/Ruby objects back and forth. You would call me crazy. You would never think of writing tests for Python code in any language but Python! Even if this was the only way to test Python code, you wouldn't use it because it would be too complex.

This is the problem I see with Selenium. JavaScript is the native scripting language for browsers and it is also the language that our application is written in. We should be writing tests in JavaScript.

The only feature that Selenium provides that single page JavaScript doesn't is the ability to have tests that span multiple pages. But I still think we can get around that using an iframe as I described above.

@ctb
ctb commented
@takluyver
Owner
@stefanv

I guess the concern is that the Python expressions for locating and exercising DOM objects may be clunky. As far as webdriver goes, however, they are very similar. Compare the WebDriverJS link above to the examples here:

http://seleniumhq.org/docs/03_webdriver.html

The Javascript route comes at some cost: no nose integration, and no automated server launch--neither that big a deal?

I agree that testing of Javascript classes and methods should be done in Javascript itself.

@ellisonbg
Owner
@stefanv
@ctb
ctb commented
@ellisonbg
Owner
@ellisonbg
Owner
@stefanv

I think it's fairly easy writing those tests, provided that all the elements on the page have appropriate IDs assigned (that's the main workaround I had to do to find certain buttons, etc.). They look very similar in Javascript and Python, so not sure the choice of language makes much of a difference.

@ellisonbg
Owner

There is a broader discussion about the testing of the notebook that needs to happen independent of this PR. I have opened an issue to track that discussion (#2292) and am going to close this PR for now. This PR may end up being part of the solution, but we are not yet to the point of making that decision.

@ellisonbg ellisonbg closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
4 IPython/frontend/html/selenium/.gitignore
@@ -0,0 +1,4 @@
+profile_default/history.sqlite
+profile_default/startup/
+profile_default/security/
+*.ipynb
View
26 IPython/frontend/html/selenium/pageobject/__init__.py
@@ -0,0 +1,26 @@
+from driver import driver
+d = driver()
+
+import urlparse
+import os
+import json
+
+_config_file = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ '../selenium.json')
+config = json.load(open(_config_file, 'rb'))
+
+port = config['port']
+uri = 'http://localhost:%d/' % port
+locators = config['locators']
+
+def find(element):
+ from driver import driver
+ return driver().find_element_by_id(locators[element])
+
+class PageObject(object):
+ def __init__(self, path):
+ self._uri = urlparse.urljoin(uri, path)
+ self.load()
+
+ def load(self):
+ d.get(self._uri)
View
5 IPython/frontend/html/selenium/pageobject/dashboard.py
@@ -0,0 +1,5 @@
+from . import PageObject
+
+class Dashboard(PageObject):
+ def __init__(self):
+ PageObject.__init__(self, '/')
View
14 IPython/frontend/html/selenium/pageobject/driver.py
@@ -0,0 +1,14 @@
+from selenium import webdriver
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.common.keys import Keys as keys
+
+_cache = {}
+
+def driver():
+ if not 'driver' in _cache:
+ _cache['driver'] = webdriver.Firefox()
+
+ return _cache['driver']
+
+def wait(condition):
+ WebDriverWait(driver(), 10).until(condition)
View
39 IPython/frontend/html/selenium/pageobject/test_dashboard.py
@@ -0,0 +1,39 @@
+from driver import driver, wait, keys
+d = driver()
+
+from dashboard import Dashboard
+from . import find
+
+def test_basic():
+ db = Dashboard()
+ assert 'IPython Dashboard' in d.title
+
+def test_create_notebook():
+ db = Dashboard()
+
+ # Click new notebook button
+ find('dashboard.new_notebook').click()
+ d.switch_to_window(d.window_handles[-1])
+
+ # Click on the notebook name to rename it
+ find('dashboard.notebook_name').click()
+
+ # Wait for the popup dialog
+ wait(lambda d: d.find_element_by_class_name('ui-dialog-titlebar-close'))
+
+ # Type in the new name
+ name_input = d.find_element_by_tag_name('input')
+ name_input.clear()
+ name_input.send_keys('Test Notebook')
+
+ # Click OK
+ old_name = find('notebook.name').text
+ buttons = d.find_elements_by_class_name('ui-button')
+
+ OK_button = [b for b in buttons if b.text == 'OK']
+ OK_button[0].click()
+
+ # Wait until save is done
+ wait(lambda d: old_name != find('notebook.name').text)
+
+# find('notebook').send_keys(keys.CONTROL + 'm', 's')
View
16 IPython/frontend/html/selenium/profile_default/ipython_notebook_config.py
@@ -0,0 +1,16 @@
+c = get_config()
+
+# Hashed password to use for web authentication.
+#
+# To generate, type in a python/IPython shell:
+#
+# from IPython.lib import passwd; passwd()
+#
+# The string should be of the form type:salt:hashed-password.
+# c.NotebookApp.password = u''
+
+# The base URL for the kernel server
+# c.NotebookApp.base_kernel_url = '/'
+
+# The port the notebook server will listen on.
+# c.NotebookApp.port = 8888
View
10 IPython/frontend/html/selenium/selenium.json
@@ -0,0 +1,10 @@
+{
+ "port": 10987,
+ "locators": {
+ "notebook": "ipython_notebook",
+ "dashboard.new_notebook": "new_notebook",
+ "dashboard.notebook_name": "notebook_name",
+ "notebook.save_status": "save_status",
+ "notebook.name": "notebook_name"
+ }
+}
View
2  IPython/frontend/html/selenium/selenium_docs.txt
@@ -0,0 +1,2 @@
+http://selenium.googlecode.com/svn/trunk/docs/api/py/webdriver_remote/selenium.webdriver.remote.webdriver.html
+http://selenium-python.readthedocs.org/en/latest/index.html
View
87 IPython/scripts/iptest-web
@@ -0,0 +1,87 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import subprocess
+import shlex
+import socket
+import time
+from threading import Thread
+import functools
+
+import nose
+
+base = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ '../frontend/html/selenium')
+os.chdir(base)
+
+sys.path.insert(0, base)
+from pageobject import port, driver
+
+
+def get_ipython_binary():
+ from IPython.utils.path import get_ipython_module_path
+ from IPython.utils.process import pycmd2argv
+ argv = pycmd2argv(
+ get_ipython_module_path('IPython.frontend.terminal.ipapp'))
+
+ return argv
+
+def log(m):
+ print m
+
+def clean_notebooks():
+ for nb in [f for f in os.listdir(base) if f.endswith('.ipynb')]:
+ os.remove(nb)
+
+def verify_server_running(port, status):
+ s = socket.socket()
+ while status[0] != 'abort':
+ try:
+ s.connect(('localhost', port))
+ s.close()
+ except socket.error:
+ time.sleep(0.2)
+ else:
+ status[0] = 'up'
+ return
+
+def launch(timeout=5, port=10987, flags=''):
+ log('Firing up IPython notebook...')
+
+ ipython_cmd = '''
+ notebook --no-browser --port=%(port)d --user="testuser"
+ --ipython-dir="." --profile="default" %(flags)s
+ ''' % {'port': port, 'flags': flags}
+ p = subprocess.Popen(get_ipython_binary() + shlex.split(ipython_cmd))
+
+ status = ['down']
+ t = Thread(target=verify_server_running, args=(port, status))
+ t.start()
+ t.join(timeout)
+
+ if status[0] != 'up':
+ log('No sign of life after %d seconds. Aborting.' % timeout)
+ status[0] = 'abort'
+ t.join()
+ sys.exit(-1)
+ else:
+ log('Server is alive.')
+
+ return p
+
+def run_nose(verbose=False):
+ args = ['', '--exe', '-w', './pageobject']
+ if verbose:
+ args.extend(['-v', '-s'])
+
+ nose.run('pageobject', argv=args)
+
+
+clean_notebooks()
+
+p = launch(port=port)
+run_nose(verbose=True)
+p.terminate()
+
+driver().quit()
Something went wrong with that request. Please try again.