Tests framework to HTML-based apps #65

Open
taboca opened this Issue Feb 15, 2011 · 6 comments

Comments

Projects
None yet
2 participants
Contributor

taboca commented Feb 15, 2011

I am working on a proposal to test HTML-based apps. This solution is an initial experimentation, based in Jetpack SDK tests framework, and may also serve to do other things in "batch mode". The general design is:

A (for HTML apps) test solution should run Chromeless a number of times, one time or many, for each test/sample application. Chromeless itself is a Jetpack-based application so its main component is located in :

[1] ./packages/chromeless/lib/main.js

Following the test framework in Jetpack SDK, we can place a test directory in:

[2] ./packages/chromeless/tests/test-main.js

And then, when using "test" execution mode ( against Python Cuddlefish cfx ,) the test-main.js should be executed. And if in [1] there is a exports.main, then it means that test-main.js can access the scope of main application. So, in theory, test functions in test-main.js should be able to execute functions over main.js, for example it should be able to launch a Chromeless-based HTML app -- default main function.

In a normal case, for a Jetpack app, the test rules ( assertions and function calls against main app ) are placed in test-main.js and are specific for each Jetpack app. In our design, however, the test-main.js would act like a proxy and perform actual tests that should be located in the actual Chromeless sample app directory. So, for example, a Chromeless app that is located in ./samples/myApp would need a file for tests -- let's say a tests.js file. This file would have a list of test functions and these functions should be able to execute calls in the scope of the main HTML app, and as well perform assertions over back to the test-main.js which in turn will do the assertions as it does with Jetpack SDK.

To get started I took Jetpack recent SDK and merged with some fundamental Chromeless patches to have it launching XULRUnner-based app and the HTML-based app. The following is work-in-progress:

https://github.com/taboca/addon-sdk/commits/

Status is:

  1. It properly executes test-main.js in ./packages/chromeless/tests
  2. It still does not launch the main.js ./packages/chromeless/lib/main.js from the test-main call
Contributor

lloyd commented Feb 16, 2011

This sounds reasonable, but I'm having some trouble getting my head around things. Here's some questions:

  1. From the command line, how will I run all tests?
  2. From the command line, how will I run one specific test?
  3. What's an example of a test that runs in "privileged" mode? (the actual javascript source)
  4. What's an example of a text that runs as application code? (the actual javascript source)
  5. How do you differentiate between the two?
  6. will existing jetpacky test run unchanged?
Contributor

taboca commented Feb 16, 2011

Thanks for the questions.

  1. The comand line could be ./chromeless testall
  2. The comand line could be ./chromeless test appname

But under the main Python's cuddlefish code, what happens is that we are calling a "cfx test" = run command ="test. So here is a snapshot of a hardcoded test script for one specific app:

#!/usr/bin/env python
import cuddlefish
import simplejson as json
cuddlefish.run([
    "test",
    "-a", "xulrunner",
    "-b", "./build/xulrunner-sdk/bin/xulrunner-bin",
    "--static-args", json.dumps({"browser": "./apps/dragdrop/index.html"}),
"--templatedir", "./app-extension",
    "--pkgdir", "packages/chromeless"
    ])

It instructs Python's cuddlefish code to run in "test" mode which will trigger the execution of another app instead what we would normally expect, our actual --pkgdir app ( our top level Jetpack SDK-based app ). The main app executed, in this mode, is

./packages/test-harness/lib/run-tests.js 

And this will execute a testing app under our package dir ( --pkgdir ), under ./tests directory. Above referred, run-tests.js, will find all the ./tests/test-* files for a given --pkgdir app.

And here is an initial test file:

https://github.com/taboca/addon-sdk/blob/3d35980a22c0662f48091504276e9984d7fef007/packages/chromeless/tests/test-main.js

Notice, looking in the above file, that testBrowser() function does call our exports.main for the ./packages/chromeless/lib/main.js. It is passing a hardcoded ( TBD fix ) filename for the actual HTML app.

3)4)5) you mean privileged mode = Jetpack app and app mode = Chromeless target app right?

This solution is going towards Chromeless target apps -- means to grab/execute tests functions described in the scope of the actual target app ( HTML/JS app ). The actual Jetpack-based app, which I think you refer as privileged mode, should be done with no changes in the Jetpack SDK I think. For pure Jetpack one you can check examples on tests if you look in ./examples/reddit-panel/tests/ for instance ( jetpack SDK ).

  1. Mostly changes to how the Chromeless/tests/test-main.js works. So to summarize this proposed solution, we are modifying what would be a normal Jetpack test file, in this case the ./packages/chromeless/tests/test-main.js to work as a mediator. So instead all the tests functions being located in this file, we will take them from the actual app directory ( ./samples or ./examples in chromeless ) and run the tests on behalf of the Chromeless Jetpack app ( main.js .)

The hardcoded case is there -- it's not yet taking the test functions from the context of the Chromeless target app ( TBD fix ). If you look in the above test-main.js, you will see that it instructs Chromeless main to start itself, it passes a browser/app filename, it waits 5 secs, then it grabs the title of the main app. It's meant to prove that it does intercept the DOM at the app level.

Moving forward:

An actual JS for tests, for a given Chromeless target app, could be:

var testsManager = require("test"); // This makes a bridge with this test-main.js
function testSomething() { 
     testsManager.assert(document.title.length>0); 
} 
testsManager.bind(testSomething);

So when said test-main.js is executed, it would execute if any ( pre-bound ) test functions. And these would perform the actual call to assert functions in the context of the test-main.js. I am moving towards now to this last part so the code that is hard coded in test-main.js, that currently access the DOM for an actual target app, will not exist in here.

Contributor

taboca commented Mar 8, 2011

Example of a test.js script file under the ./tests/close/ directory:

SHA: 50a6bf3

In this example, we dispatch a click to the button close, and we check if the appWindow is present or not, so the test passes if the appWindow ( getAppWindow ) object is null.

Contributor

taboca commented Mar 17, 2011

An up to date proposal patch is here:

https://github.com/mozilla/chromeless/tree/dev_merge_sdk
Specifically :
42ab013

Contributor

taboca commented Mar 29, 2011

We have now, landed, our initial tests framework. The idea is basically a reuse of Jetpack SDK test-harness, except that we are not using the run_app approach for Mac OS X. Instead, for Mac OS X only, we are using appify approach which creates a local standalone app and simply does open the app that was created in the hard disk.

So, now, we will need figure out how to pass a new profile for every iteration. I think this is important because we don't want to run tests with always the same default profile that was created by a given XULrunner app. By the way, when running XULRunner app with the .app / Mac OS X open, XULRunner makes its decision to create the profile or reusing existing. And it creates in a place like this:

/Users/user_name/Library/Application Support/My Chromeless App/Profiles/6y09zele.default/

On the other hand, when calling XULRunner using command line, the normal approach to use profiles is via arguments:

xulrunner-bin -profile path_to_profile

There are ways to tackle this:

  1. one is to use Jetpack SDK approach = run_app for tests; Which uses xulrunner-bin and has already everything to create new empty profiles.

  2. We put a way for appify to create a new random app name using a random ID at the end of the file name. And we woudl need to make sure we do the maintenance.

  3. Maybe we could pass the -p parameter in info.plist file? Not sure.

@ghost ghost assigned taboca Apr 18, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment