Skip to content

Yeti Next

Jump to bottom
reid edited this page Nov 17, 2011 · 11 revisions

As existing open source testing solutions prove inadequate, Yeti will begin to compete in the Continuous Integration space.

What is Yeti Next?

An effort to develop Yeti into a better tool for browser testing that integrates well with CI and other systems.

Problems to Solve

After discussions with our customers, a few problem spaces have been identified:

  • Commanding other browsers to open and close Yeti's test page.
  • Exporting test results.
  • Stateless server.
  • Integration with (mocked) server-side applications.
  • Ability to work with other browser test frameworks.
  • Ability to test server-side JavaScript.

Constants to Maintain

Yeti does a few things very well. These things must be maintained:

  • Simplicity.
  • The existing command-line interface.
  • Ability to run browser tests quickly by "capturing" a browser.
  • Speedy test execution using the socket.io system for browser and internal communication.
  • Mobile browser compatibility, including Android and iOS.

Problem Priority

The problem areas are being addressed in order of priority.

  1. Yahoo!: Integration into Node.js-based browser launcher. Easy access to test results.
  2. Canonical Ltd.: Easy access to test results.
  3. YUI Team: Replacement of YUI Test Selenium Driver and possibly Selenium itself.

Our releases will be timed to maintain this priority order; however, the design changes made will take all of these things into account.

Key Concepts

Client

A Client provides code that needs to be tested. It uses the Yeti Client API. This may include:

  • The Yeti Command-Line Interface.
  • A component of an automated CI system.
  • A plugin for another test framework tool.

All Clients provide:

  • An HTTP server, started through the Yeti Client API, that serves code that will be tested.

Agents

An Agent is anything that will run Yeti tests. It speaks the Yeti Agent Protocol. This may include:

  • Yeti's "capture" page inside a browser.
  • An app that manages browsers in a VM Lab. (See Browser Control System below.)
  • An app on a mobile device that tests the native browser, UIWebView, etc.

In the future, any program that speaks the Yeti Agent Protocol can run a Yeti test. This may include:

  • Node.js testing with Vows or YUI Test.
  • Ruby, Python, Perl, etc.

Hub

A Hub is a daemon that brings Yeti Clients and Agents together.

You may run a Hub:

  • By running yeti --server.
  • By using the Yeti Hub API. (Future task.)
  • As a part of a larger test framework or CI system.

A Hub has:

  • 0..n Agents that it maintains.
  • 0..n Clients that request use of the Agents.

SuperHub

A SuperHub is a collection of Hubs. To the Yeti Client API, a SuperHub appears identical to a Hub.

This concept is similar to the notion of the Selenium Hub; however, unlike Selenium there's no difference between a "hub" mode vs. a "standalone" mode. The Yeti Hub API will provide a way to connect a new Hub to an existing one, at which point the existing Hub will provide the Agents of both to its Clients.

API Development Priority

As of November 2011, the Client API described below will be developed first, according to this order.

  1. Client API. (Includes the test-serving HTTP Server.)
  2. Hub API.
  3. Agent API / Yeti Agent Protocol.

Aside from the Client API, the other APIs are not yet designed or scheduled. Private APIs will be used until time is allocated for developing public versions.

Yeti Client API (November 2011 sprint)

Nothing throws.

	var yeti = require("yeti");

	// Likely socket.io, not WebSocket, but you'd pass a URL here.
	// If nothing is provided, you could also start a new Hub.
	// For now, Hubs are started with `yeti --server --port 9990` for example.
	var hub = yeti.hub("ws://yeti.corp.yahoo.com:9990/");

	// The name is set by the Hub. (For Dav!)
	assert.ok(hub.name == "YUI Yeti Hub");

	// Disconnect if you're done:
	// hub.disconnect();

	hub.agents();
	// -> [{id:"12345", platform: "mac", browser: "chrome", version:"15", connected: true}, {id:"94111", platform: "windows", browser: "internet explorer", version: "9", connected: true}]

	hub.agent("12345").on("disconnect", function () {
		console.log("Chrome disconnected, oh no!");
	});

	// This is a function call largely because it keeps the same API
	// as `hub.agents(…)`, and I don't want to keep the property
	// in sync across lots of agents.
	assert.ok(hub.agent("12345").connected());

	hub.on("agentConnect", function (agent) {
		console.log("New Agent connected and ready to use: " + agent.id);
	});

	hub.on("agentSeen", function (agent) {
		console.log("New Agent is hopefully starting up: " + agent.id);
	});

	var server = yeti.server({
		docRoot: "/Users/reid/Development"
	});

	// This is an http.Server, so it's pretty easy to extend.

	server.listen(8000);

	// Finally, run tests:

	var someAgents = hub.agents("12345", "94111");

	assert.ok(someAgents.connected()); // Both are connected.
	assert.ok(someAgents.available()); // Both are available. An agent is unavailable if it's running a batch. If an agent isn't avaialble when a new batch is dispatched, then the new batch will start when it becomes available.

	var batch = hub.dispatch({
		agents: someAgents, // really just an 'array' of agent info
		tests: server.tests(["yui3/src/widget/tests/widget.html"])
	});

	// Or maybe instead:
	/*
	This was not chosen, because it should be possible to queue
	tests before browers connect? Or maybe that's a bad idea,
	and this API is a good one:
	var batch = someAgents.dispatch(
		server.tests(["yui3/src/widget/tests/widget.html"])
	);
	*/

	someAgents.on("disconnect"); // One of the agents in someAgents has disconnected. Maybe halt?

	// If an agent in a batch disconnects, but reconnects before the batch's end event fires,
	// it will be re-added to the batch and will continue from the last test it reported.
	// Nothing special will happen if the agent conencts after the end event fires.

	// batch.halt(); // Stops everything and reset agents back to idle state.
	// Pass true to hard-reset: Yeti's Browser Agent will redirect to a new page first to clear GC.

	batch.on("data"); // Subscribe to tests as they are submitted to Yeti's Hub.

	batch.on("end", function (err, results) {
		// Records not only aggregate results, but which agents completed them (they may have dropped off)
		if (err) {
			// Static test validation failed, no agents were connected, fatal error, etc.
			// An error will never occur for agents being unavailable, we would simply wait for them.
			console.log("Something went wrong: " + err);
			// TODO: What should `err` be? A code?
		} else {	
			console.log("All clients have completed testing this batch.")
		}
	});

	batch.agents().on("end"); // Fires as each agent completes testing.

	batch.agent("94111").on("end", function (err, results) {
		if (!err) {
			console.log("Internet Explorer is done with this batch without any fatal problems.");
		}
	});

	batch.agent("94111").halt(); // Just stop IE.

	batch.agent("94111").on("data"); // Just IE test data.

Local Only

	var yeti = require("yeti");

	var hub = yeti.listen(8080);

	// Capture browsers at http://localhost:8080
	
	var batch = hub.dispatch([
		"src/widget/tests/widget.html",
		"src/widget/tests/widget2.html"
	]);

	// When the hub is localhost, files are served
	// by a `yeti.server` that's built into the hub.
	// It uses process.cwd() as the docRoot.

	batch.on("end", function (err, results) {
		if (err) {
			throw err;
		}
		console.log(results.passed + " tests passed");
	});

If we wanted more control:

	// Note: agent, singular.
	// We don't need to test every iPhone we have connected,
	// just an available one.

	hub.dispatch({
		tests: testPaths,
		agents: hub.agent({platform: "iphone"})
	});

	batch.on("end", function (err, results) {
		if (err) {
			console.log("Whoops, probably no iPhones here.");
		}
	});

Browser Control System

In order to more efficiently use Yeti with a CI system, a script using the Yeti developer API should be able to command browsers to open to Yeti's test page. Possibilities include:

  • Integration with Soda, the old-school Selenium RC API for Node.js.
  • Contributing WebDriver additions to Soda.
  • Developing a lightweight agent program to open and close browsers for iOS and desktop browsers.

If Selenium integration is chosen, existing Selenium solutions are inadequate for running for the YUI team's own use. Better solutions would include:

  • Simpler software. Ability to hit a URL, open an executable and "capture" the entire system as a Yeti slave.
  • BrowserBox, Yeti edition. A Linux VM that contains many different browsers with software pre-installed that make it work with Selenium or Yeti.
  • Selenium-as-a-service. Internal projects, SauceLabs, etc. that provide a Selenium Hub as a service.

The Browser Control System should be assigned to another person to be worked on in parallel to Yeti work.

Backlog

  • User-facing interface remains unchanged.
  • Developer API. require("yeti") and script your own additions.
    • Provide the ability to "plug-in" to various parts of Yeti's lifecycle.
    • Export test results using this system.
    • Events will be used, see API section below.
  • Integration with Browser Control System of choice.
  • Ability to replace YUI Test Selenium Driver in our own CI system.

Technical Information

High-level

  • Everything in socket.io, it's fast.
  • Browser next() - must be cross-domain, to make it work with proxied requests out.

Documentation

  • Publish "Best Practices" for setting up a browser for Yeti testing.
    • No lock screen on iOS
    • IE checkboxes: disable script dialogs
    • Flash transport for older IEs
    • Firebug disabled not present
    • No overlap of OS windows (focus tests)

Yeti Built-In Browser Agent

No iframe? It breaks some tests.

  • Halt / Reset. Easy to terminate all tests running, in a batch.

    • Abort: Stop
    • Reset: Redirect to empty page, then restart (clear GC) -- also clear session cookie

  • Reset / redirect GC clear ON MOBILE before next batch.

Mobile Testing

Support UIWebView and MobileSafari testing, since they are different.

CLI

  • Disable passing test output.
  • Name of Hub connected.

Original Notes

Here are the original notes for the API section.

Agent Management on the Hub

These APIs will use a unique ID per Agent during Yeti's server lifetime. They also provide a browser's User-Agent and other metadata, if appropriate.

  • Agent Seen event, if successfully loaded, you'll get:
    • Agent Connected event.
  • Agent Disconnected event.

When a Agent Seen event occurs, the Agent will be assigned a unique identifier that persists until the Agent is stopped.

This identifier is used to identify the Agent. (Note: socket.io identifiers will not be used.)

These events are provided on the Hub.

Batch Management

These APIs are for managing tests.

  • Batch added. Perhaps per device. Fire with testfiles:
    • Sanity check first. Fire end with error if this fails, including bad testfiles.
    • On a browser reconnect, resume tests in the same spot. Keep track of the last test ran.
  • Batch complete. Per device. This API provides device information.

This API will return test results.

  • Batch completed - for all devices.
  • Batch completed - for a browser ID.

Yeti

These APIs are for managing Yeti itself.

  • Add your own routes. (Expose Connect Express Router.)
Clone this wiki locally