SageNB -- Add a Selenium test suite.

[TimDumol]

Inclusion of a test suite will prevent regressions, and make development much easier.

CC: @williamstein @qed777

Component: notebook

Author: Mike Hansen, Tim Dumol

Reviewer: Mitesh Patel

Issue created by migration from https://trac.sagemath.org/ticket/7343

[TimDumol]

New Selenium test suite.

[TimDumol]

comment:1

Attachment: trac_7343-selenium-tests.patch.gz

Depends on #7309, #7310 and #7332. This is an initial version of the test suite. Many of these test cases were adapted and mildly modified from Mike Hansen's original Selenium test code.

More test cases to come, but I feel that having a test suite included asap is essential.

[TimDumol]

Author: Mike Hansen, Tim Joseph Dumol

[TimDumol]

comment:2

Running the test suite:

sage: from sagenb.testing.run_tests import run_tests; run_tests()

Note that this requires an instance of the Selenium server running on port 4444. The Selenium server can be downloaded from http://seleniumhq.org/download/ as part of the Selenium RC package and can be run with java -jar selenium-server.jar. This could not be included in the patch due to possible conflicts with the system binaries and libraries (Firefox, etc.).

[TimDumol]

comment:3

A note: There is a failure in test_searching_worksheets. It is an actual bug -- the requisite javascript libraries are not loaded.

[TimDumol]

The previous patch had some errors (*~ files, missing file).

[TimDumol]

Attachment: trac_7343-selenium-tests.2.patch.gz

Attachment: trac_7343-selenium-tests.3.patch.gz

Added a test to confirm that #7341 works.

[TimDumol]

Added "Edit worksheet" test. Made sure that no orphan processes are left behind by failed tests.

[qed777]

comment:4

Attachment: trac_7343-selenium-tests.4.patch.gz

So far, this looks great! I'm still reading through the Selenium-RC documentation, but I think I can review this ticket formally tomorrow. At the risk of asking too soon: Can we specify the particular browser to test in run_tests()? Another potentially useful option, at least for Firefox: Use, e.g., firefox -P selenium-test -remote "openURL($*, new-tab)", to open multiple tabs instead of windows. I'll investigate.

Aside: It's nice that both Selenium and FunkLoad are build on Python's unittest framework. Of course, FunkLoad only simulates a browser, but I think we can attain approximate parity between the notebook's functional (i.e., with Selenium) and load (i.e., with FunkLoad) tests. Selenium's advantage is that it tests real-world browsers, but even with Selenium-Grid, we'd have difficulty (I think) simulating thousands of simultaneous notebook users.

[qed777]

comment:5

Possibly useful: HTMLTestRunner.py?

[qed777]

Tweaked a few tests. See the comment. Apply only this patch.

[qed777]

Reviewer: Mitesh Patel

[qed777]

comment:6

Attachment: trac_7343-selenium-tests.5.patch.gz

Version 5:

• Fixes TestWorksheet.test_edit.

• Changes TestWorksheet.test_7341, for now, to use

sel.get_eval('window.evaluate_cell_introspection(1, null, null);')

instead of

self.tab('cell_input_1')

For some reason, self.selenium.key_press_native(9) doesn't consistently send a TAB character in Firefox 3.5.3, at least for me. I think the problem is Selenium's.

• Simplifies TestWorksheetsList.test_searching_for_worksheets. Is searching published worksheets broken in sagenb's tip? I see, e.g.,

        exceptions.IOError: [Errno 2] No such file or directory: '/home/.sage/sage_notebook.sagenb/home/pub/12/worksheet.html

if I publish a worksheet, visit "Published," and try a search. Restarting the server fixes the problem.

Anyway, this is great work! To the extent it counts, my review is positive.

[qed777]

comment:7

A sample Selenium-RC startup script:

#!/bin/bash
JAVA="/usr/java/latest/bin/java"
SEL_DIR="/home/apps/selenium/selenium-remote-control-1.0.1/selenium-serve
r-1.0.1"

FF_BROW="*firefox3"
FF_EXEC="/usr/lib64/firefox-3.5.3/firefox"
FF_PROF="/home/sage/selenium/firefox/selenium1"
OP_BROW="*opera"
OP_EXEC="/usr/lib/opera/opera"
CR_BROW="*chrome"
CR_EXEC="/usr/lib64/chromium-browser/chromium-browser"

BROW="$FF_BROW"
BROW_EXEC="$FF_EXEC"

OPTS=""
#OPTS="$OPTS -firefoxProfileTemplate $FF_PROF"
OPTS="$OPTS -singleWindow"
#OPTS="$OPTS -browserSessionReuse"
#OPTS="$OPTS -ensureCleanSession"
#OPTS="$OPTS -debug"
#OPTS="$OPTS -browserSideLog"
#OPTS="$OPTS -log test.log"
#OPTS="$OPTS -trustAllSSLCertificates"
#OPTS="$OPTS -forcedBrowserMode $BROW"
OPTS="$OPTS -forcedBrowserModeRestOfLine $BROW $BROW_EXEC"

$JAVA -jar "$SEL_DIR/selenium-server.jar" $* $OPTS 

Note: Opera 10 and Chromium are not supported.

[qed777]

comment:8

Replying to @qed777:

Possibly useful: HTMLTestRunner.py?

See #7390.

[williamstein]

comment:9

Unfortunately, I can't use any of this under OS X, because the sqlite libraries included in OS X and Firefox are incompatible:

• http://jira.openqa.org/browse/SRC-743

• https://bugzilla.mozilla.org/show_bug.cgi?id=513747

After nearly two hours of trying to get this to work, I'm giving up for now. I guess Selenium is just broken on OS X for now. Hopefully the Firefox and/or Selenium devs will fix this asap. In the meantime, this is a vote from me to check out FunkLoad: "Of course, FunkLoad only simulates a browser, but I think we can attain approximate parity between the notebook's functional (i.e., with Selenium) and load (i.e., with FunkLoad) tests." If nothing else, it would be really good to have a test suite that we can include and that anybody can trivially use with no confusing (or in my case, impossible) setup and configuration issues.

[williamstein]

comment:10

Further comments about this:

(1) I tried to use FunkLoad to test Sage (again in OS X) for a while, and got nowhere. It seems to be a mess and not very polished. Evidently it exists because some company wrote it and thought it might be useful to the community, so they released it. I'm dubious about FunkLoad.

(2) No matter what, I think we need functional testing of the sage notebook that works on our build farms. There's no way Selenium can do that. I think we need something else to complement Selenium, possibly just something straightforward written in python using urllib, urllib2, etc.

[qed777]

comment:11

Some browser drivers and/or simulators:

• FunkLoad simulates.
• Pylot simulates.
• Sahi drives.
• Selenium RC drives but see WebDriver below. Selenium Grid drives Selenium RC.
• Watir drives, but Celerity simulates.
• WebDriver is merging with Selenium. See this blog post and the FAQ. It drives and simulates. Already?
• Windmill drives.
• zope.testbrowser simulates.

I apologize for any misclassifications. Some of the simulators can simulate JS, to a degree. I think these are all based on HtmlUnit.

It would be great if we could write tests in just one actively developed and supported framework/API but be able to run both truly functional and simulated tests. Selenium/WebDriver may be(come) that framework.

[qed777]

comment:12

By the way, this ticket depends on #7309, #7310, and #7332.

[qed777]

comment:13

The Grinder, a Java load testing framework, has Jython bindings. It might be worth a closer look.

[qed777]

comment:14

Notes:

• I tried testing WebDriver's Python bindings to no avail. It may be best to wait for a post-merger release of Selenium.

• Pylot test cases must be written in XML, which is not as expressive as Python.

• Can we use Selenium Grid with a Linux virtual machine to work around the OS X problem, for now?

• zope.testbrowser seems promising for light-weight functional testing.

• For load tests, I think The Grinder and (yes) FunkLoad are worth a closer look. To get some data, I'll try to extend the simple FL test I ran recently to more complicated scenarios. The Grinder may well be a better choice, although it's written in Java, since it seems to be mature, well-documented, and actively developed. Moreover, we can write tests in Python, more or less.

[qed777]

comment:15

Good news, I think: We can use FunkLoad to create a new worksheet, enter/evaluate a cell, and check for updates every 0.25 seconds. We just need to send a [conditional] sequence of HTTP GET and POST requests. Unless there are objections, I'll try to carry this further, along the lines of the Selenium test suite. To the extent that FL is pure Python, its "functional" and load tests should run on multiple platforms.

Note: I did need to make one change to FL's redirect handling code:

--- trunk/src/funkload/FunkLoadTestCase.py      2009-11-11 02:31:27.000000000 -0800
+++ funkload/src/funkload/FunkLoadTestCase.py   2009-10-14 14:51:19.000000000 -0700
@@ -277,6 +277,9 @@ class FunkLoadTestCase(unittest.TestCase
             thread_sleep()              # give a chance to other threads
             while response.code in (301, 302, 303, 307) and max_redirect_count:
                 # Figure the location - which may be relative
+                cookie = response.headers.get('Set-Cookie')
+                if cookie:
+                    self.setHeader('cookie', cookie)
                 newurl = response.headers['Location']
                 url = urljoin(url_in, newurl)
                 # save the current url as the base for future redirects

[qed777]

comment:16

By the way FL, uses gnuplot to make graphs for bench reports. I suppose we could use matplotlib or Sage, instead, but I haven't examined the plotting code.

[TimDumol]

comment:17

Replying to @qed777:

Good news, I think: We can use FunkLoad to create a new worksheet, enter/evaluate a cell, and check for updates every 0.25 seconds. We just need to send a [conditional] sequence of HTTP GET and POST requests. Unless there are objections, I'll try to carry this further, along the lines of the Selenium test suite. To the extent that FL is pure Python, its "functional" and load tests should run on multiple platforms.
[snip]

Noting that FunkLoad apparently requires patching before it is fully usable, what are the disadvantages to using zope.testbrowser, which seems to be better maintained and planned? I haven't tried it personally yet though, so I my impression may be wrong.

[qed777]

comment:18

I also haven't used zope.testbrowser, but after reading through its documentation, I think it's better for "functional" testing than for load testing. In particular, it [very likely] has better cookie and form-handling capabilities than FL (actually, webunit). On the other hand, FL already has an infrastructure for concurrent testing and it can load/cache CSS/images/etc.

Of course, neither is a replacement for Selenium's truly functional tests. For "functional" tests alone, z.t may be a good choice. For load tests, FL seems to be a good Pythonic choice that can also go "functional." However, if we don't mind using other languages, Watir/Celerity or The Grinder may be better. How important are the load tests? Perhaps we can build a load test framework on z.t?

[TimDumol]

comment:19

I would think that load tests are quite important, since http://sagenb.org serves in excess of 10k users. I personally don't mind using other languages.

[williamstein]

comment:20

By the way, this ticket depends on #7309, #7310, and #7332.

Thanks for pointing this out. The sagenb-0.4.1 release I made that was going to to go into sage-4.2.1 contains #7343 but not #7309, #7310, and #7332. I'll merge those.

[qed777]

comment:21

"If nothing else, it would be really good to have a test suite that we can include and that anybody can trivially use with no confusing (or in my case, impossible) setup and configuration issues."

To achieve this --- moreover, a fast, self-contained suite --- Python is a logical choice. But setup/config may be significantly simpler with simulated browsers. I suppose we need to cover three areas with up to three frameworks:

a. In-browser - Se, Wa.
b. Sim-browser - FL, Se 2.0, Wa, Zt.
c. Load - FL, Gr.

[qed777]

comment:22

Replying to @williamstein:

By the way, this ticket depends on #7309, #7310, and #7332.

Thanks for pointing this out. The sagenb-0.4.1 release I made that was going to to go into sage-4.2.1 contains #7343 but not #7309, #7310, and #7332. I'll merge those.

For 4.2.1, it may also be worthwhile to include one or more of these bug fixes: #7316, #7318, #7339, #7354, #7385.

[williamstein]

comment:23

I've merged this into sagenb-0.4.2 (sage-4.2.1)

[qed777]

comment:24

A passing thought: We could reuse some functions in a "functional" Python test suite for a Sage Remote Access API (see, e.g., Google Documents List Data API). This would permit authenticated, programmatic access for manipulating worksheets. This may be an independent reason to use Zt.

[qed777]

comment:25

On organizing tests:

Suppose we put all framework-dependent code (or as much as possible) in sagenb.testing.notebook_test_case, where we define NotebookTestCaseSelenium, NotebookTestCaseZopeTestbrowser, etc., as subclasses of NotebookTestCaseAbstract. Then, we can write tests under sagenb.testing.tests with a [largely] framework-independent, higher-level API. We can select a framework with which to run the tests in sagenb.testing.run_tests.

Of course, we can add specialized methods for particular frameworks. Moreover, over time, we can add or remove framework classes without discarding too many tests.

Thoughts?

[fchapoton]

Changed author from Mike Hansen, Tim Joseph Dumol to Mike Hansen, Tim Dumol