Intern 4 pre-beta review #732
Comments
Intern ReferenceLifecycle
Intern Facts
Use Cases
Take AwaysWe need a plugin patternWe need an easy/standard way to extend the functionality of Intern. I would prefer to have an explicit plugins configuration to explicitly load plugins during a plugin phase. As part of this pattern, it would be nice to also be able to explicitly state in what environments the plugin is loaded (Browser/Node/WebDriver) and if the default loader should be used (Node/script injection) or if the plugin requires/expects to use the configured loader. Writing a custom reporter for the browser is hardSince reporters are loaded before a loader is available it adds complexity to the user experience by requiring the user to write a Reporter that works with script injection. It makes using external libraries that require a loader more difficult to use. I think since reporters and plugins are fundamentally identical we could use the same pattern for reporters as we would for plugins. Allow reporters to be loaded via default loaders or post loader. Async preload scriptsIt looks like if I wanted to execute an async preload script, I could listen for beforeRun and return a task in my event handler. That feels a little complex. If we moved to a plugin architecture it seems like we could make this experience better. Multiple test types will pollute the rootI like intern.json and I see that the intern cli can load alternative configurations. If a user wants to have separate tests for integration, functional, unit, system, benchmark, visual regression, etc… they need multiple intern-.json files. In Intern 3, the work-around was to add code to intern.js to select the correct set of suite based on command-line parameters or produce multiple tests/intern-.js files. Configs are an issueI think we should revisit #669 and find solutions that simplify the configuration and address the issues we discussed. I’d like to know how users will
It seems like bin/intern is a temporary solution and we can address these in theintern/intern-cli. ConclusionOverall Intern 4 marks a massive improvement over Intern 3. Simply removing AMD as a dependency and supporting arbitrary loading mechanisms via a standard loader interface will make Intern more flexible for those responsible for configuring their testing framework and more familiar for those who need to use it. In a day, I haven't had a chance to do a deep code review, but from what I've seen the architecture is very good and the problems left to solve have more to do with streamlining the user experience (i.e. plugins, configuration, documentation, make it easier to load things) than making major changes. Well done! |
|
Excellent review, Paul -- thanks! Some questions/thoughts/points for discussion: PluginsDid you have any specific thoughts on what the plugin pattern might look like? Given that plugins may be loaded via script injection, they'll need to self-register ( intern.registerPlugin({
start() {
},
stop() {
}
});I'm a bit leery of having plugins rely on an external loader vs being standalone bundles. One of the main benefits of removing the loader from Intern is that Intern no longer affects the configuration or loading of an app, and it has a minimal impact on the runtime environmnent. A user could very easily end up in a situation where different plugins need different loaders (or different versions of the same loader), each with its own configuration, or a plugin and the app being tested may each require a different loader. Async preload scriptsWithout a loader, the code registration process needs to be managed (to some extent) by the script being loaded rather than by Intern. For something like registering a bit of async code, a script will need to call something on Intern to tell it that there's code to run rather than relying on Intern to manage the process. That's not to say Intern can't provide more obvious or standardized hooks. For a while I had I'm not sure how a plugin architecture would make much of a difference in this particular case. At least, given how I envision a plugin architecture working, you'd be doing intern.registerPlugin({
start() {
// do stuff
}
});vs intern.on('beforeRun', () => {
// do stuff
});Test types and configurationsI'm still not sold on the idea of building the concept of arbitrary test "types" or "stages" into Intern; it just feels too high level and use case-specific. These are inherently project and organization-specific concepts, and the features used to determine types or stages may differ between groups. For example, one org may want to separate benchmark tests from unit, which can both run in the same executor but use different reporters. Another org may want to group some visual tests in with their Node-based unit tests, which would require different executors and different reporters. In some cases, separate configs may be sufficient to encapsulate different scenarios. In other cases, something even higher-level may be required, like a user script that runs multiple runs of Intern and collects the results. I think core intern should provide basic tools that make doing something like this easier, but it shouldn't try to push users into any particular organization. The simple JSON config files actually provide a pretty decent mechanism for this. A user can setup a base intern.json, and then create separate configs that extend it. A config will encapsulate all the settings required for a particular stage (reporters, suites, pre-run scripts, etc.). I'd like to keep the config format reasonably simple; ideally it should just mirror the internal Executor config object. We could allow for multiple configs to be stored in a single intern.json file rather than requiring separate config files, though, maybe something like (where each sub-config uses the same format as the overall config): {
"suites": [ ... ],
"benchmark": false,
...
"configs": {
"browser": {
"loader": "dojo",
"suites": [ /* some other suites */ ],
...
},
"visual": {
"reporters": [ "visual" ],
"suites": [ /* visual suites */ ],
...
}
}
}Regarding your specific questions about configs:
|
PluginsMy thought behind plugins is we should formalize common patterns and encourage good usage of those patterns. I anticipate there will be two types of plug-in usage
If you're concerned about plugins having access to a loader, don't be -- it's already too late 😂! We can't stop someone from writing a preload script that waits for the loader and then loads the rest of the plugin using a loader, or worse, loading a loader in the preload scripts, or using the loader.script configuration to load a loader that then loads their stuff. We're better off defining proper patterns before users come up with creative solutions. For the first use case we should make it easy to augment Intern locally and for the second we should create a document on writing plugins, provide a recommendation for a bundler (i.e. webpack/browserify), and provide a plugin boilerplate to get people started. As far as what it would look like, plugins could replace the preload scripts in the config and could be initialized as part of the intern lifecycle. // synchronous code can run here
intern.registerPlugin({
name: string;
description: string;
metadata: any;
init(intern: Executor): Task<any> | void {
// called immediately after preload phase
}
load(intern: Executor, loader: Loader): Task<any> {
// called after the loader has been loaded
}
});
I think this addresses the first three take aways listed above. It provides a formalized plugin architecture and further documentation and boilerplate would provide good patterns for writing redistributable reporters and plugins. It provides an easy way for users to augment intern inside of their project. It makes for an easy way to do async preload scripts with I have some ideas for the configuration section. I'll post them later when I have some more time :). |
Are there any particular augmentations that are currently hard (or impossible) to implement that should be easier, or is it just that the APIs for doing so should be more centralized? I mean, a plugin API doesn't really do anything to make local augmentation easier, it seems to be focused more on redistributable code.
This looks reasonable, although I would go with something like
Plugins can certainly access the loader if a user has set one up, that's not my concern. The issue is that redistributable plugins should not be able to mandate a particular loader and/or loader config because this could become a pain point for end users. Plugins should be self contained, at least from a configuration standpoint; adding a plugin should not require a user to setup a different loader or modify their existing loader config (beyond adding a package or some such). This may require that complex plugins be distributed as bundles, but this is desirable since it means a user can just drop in a plugin without having to worry about it disrupting their testing setup. |
Right now, someone needs to have pretty deep knowledge of the event system and intern lifecycle if they want to load an async preload task or wait for the loader to be registered (listening on the Adding a
This works for me. The intention is that it'll run before tests start and after intern is fully initialized.
Agreed. What I was saying is we can only encourage good practices and should do so by supplying documentation, boilerplate, and support to plugin writers. We should also support ad-hoc plugins that allow a user to augment the system without requiring a bundler step. I wouldn't want us to remove AMD as a requirement only to make bundling another pain point. |
It's not that bad 😄 I mean, there are really only three attachment points that are available (outside of reporters): before anything runs (the intern config), before testing starts (config.before), and after testing ends (config.after). Preload scripts replace the configurability of the intern config, and the 'beforeRun' and 'afterRun' hooks handle config.before and config.after.
True.
There's nothing preventing ad-hoc plugins, so long as plugin loading happens after any loader is loaded (otherwise plugins will have to be built or not load external modules). For consistency, we should probably say that plugins will always be loaded after the loader just so there aren't 'preload' plugins and 'normal' plugins, etc. We'll still need preload scripts in case anything needs to happen before a loader is loaded, but any other hooks can be centralized into a plugin interface. Plugin functionalityWhat specific functionality should plugins provide? It sounds like right now the goal is just to provide a more obvious place than preload scripts to register 'beforeRun' and 'afterRun' handlers. Also, what should the hooking mechanism be? (Something like Reporters currently use, or something else?) In Intern 3, the closest thing to a plugin we have are reporters. They're managed by a ReporterManager, which acts as an event hub. When an event is received by the manager, it calls methods on any registered reporters. For example, when a testFail event is received, the manager calls the Intern 4 attempts to standardize things a bit more around events. Instead of having multiple ways of hooking into Intern, there's only the event system. Any code can listen for any event. There's some infrastructure for writing reporters, but at their core reporters are just event listeners (a reporter can just be a preload script with a bunch of Rather than adding the complexity of another API, it'd be nice to standardize on something that could be used both for reporters and other plugins. Currently reporters are classes that can be configured and self-register for intern events. class Foo extends Reporter {
constructor(config?: ReporterConfig) {
// ...
}
@eventHandler()
testStart(test: Test) {
// ...
}
// ...
}
intern.registerReporter('foo', Foo);The intern config declares which reporters will be used via the reporters property, which has the canonical form of: reporters: [ { reporter: foo, options: {...} } ]Reporters are loaded and registered, and then the set of reporters requested by the Intern config is instantiated in Assuming plugins will just be event listeners, the basic structure could be the same. We could say that reporters are a special case of plugin, and replace the If we want plugins to include additional metadata, the registration function could accept some sort of descriptor object instead of a name and class, like: intern.registerPlugin({
name: 'foo',
metadata: {...},
plugin: Foo
});I'm not really sure of the value of any metadata beyond the name, though. Plugins will presumably only be used when a user includes them in an intern.json, in which case the user would know what a plugin was for. Loading pluginsThere isn't currently a mechanism to load reporters through the Intern config; Intern assumes that reporters are already loaded and registered by the time the 'reporters' config property is handled. If we want the config to specify which plugins should be loaded as well as which ones should be used, we may want to do something more like: plugins: [ { name: 'foo', script?: 'path_to_foo.js', options: {...} } ]Intern could scan the plugin list for descriptors with a 'script' property and load them (which would register the plugin classes), then later instantiate the plugins. The loading process might look like: |
|
I love this. It goes a long way to normalize using external code. I have some questions. Can we leverage this same pattern for If How can users run an async preload script or plugin? I'm trying to imagine what the recommended pattern will be in the documentation. Is there a Assuming a user wants to write an ad-hoc plugin that uses their loader, it would look something like: const task = intern.loadScript('path/to/module.js');
intern.once('beforeRun', () => task);maybe we can just replace the need to write this code by adding a |
Which pattern? If we make plugins the standard place to do things, we should probably reduce the scope of what preload scripts are used for. The original intention for preload scripts was simply to let Intern load scripts like
I was thinking it would load using the currently configure loader; otherwise you'd miss out on being able to write local plugins without building/bundling them, or you'd always have to explicitly invoke loader (call
Intern's event handler waits for listeners to resolve when processing an event, so returning a Promise from a handler (like 'beforeRun') is sufficient. If Promises aren't available, the executor provides a
Currently, if a user wants to write something that uses their own loader, they would write a preload script that looked something like this (assuming, for example, their loader was SystemJS) and include it in the intern.on('beforeRun', () => {
return SystemJS.import('my/module').then(module => {
// do stuff with module
});
});The only reason we'd need to actually load the script itself with an external loader is if the script imported other modules and wasn't bundled. import module from 'my/module';
intern.on('beforeRun', () => {
// do stuff with module
}); |
|
I have more thoughts on this. Great thoughts. The greatest. |
|
Lol. Me too. Let find a day to meet and talk |
|
After some thought, how about this runtime model:
Plugins will be scripts (not modules), although those intended to run only in a Node context can assume the presence of the Node loader. (Plugins will not be able to whether they're Node-only, at least not programmatically.) Plugins can make use of an external loader in a callback that occurs after any loader has been initialized (such as a 'beforeRun' callback) if they know one will be available. Plugin APIThe ability to run code directly in a script, and to register callbacks with To really encourage the use of a intern.registerPlugin('foo', (hub, config) => {
hub.on('beforeRun', () => {
// do async thing;
});
hub.on('testEnd', () => {
// check for something
});
hub.on('afterRun', () => {
// do thing
});
});
// intern.on would no longer be availableExternal resourcesIf plugins want to bring in external resources, they'll have a couple of options. Global resources can be loaded using the built-in script loading facilities (assume our 'cucumber' module/script provides a 'cucumberFactory' function): intern.registerPlugin('cucumber interface', (hub, config) => {
return intern.loadScript('cucumber_global.js').then(() => {
hub.registerInterface('cucumber', cucumberFactory);
});
});Another option is to use Node require semantics and simply run import { cucumberFactory } from 'cucumber';
intern.registerPlugin('cucumber interface', (hub, config) => {
hub.registerInterface('cucumber', cucumberFactory);
}); |
|
Hmmm...the more I think about the idea of plugins, the more they end up looking like preload scripts. The simplest and most user-friendly option may just be to document the existing functionality (possibly renaming 'preload' to 'plugins') rather than creating an additional layer of API. |
|
We've made a lot of good progress. I think we're having trouble resolving what plugins should be is because we have a number of use cases that we're using to identify functionality that may require multiple solutions beyond just plugins. Here are the use cases and patterns we need to address Async tasksAs a plugin writer, I want to write a plugin that performs an asynchronous operation before intern continues on to the next operation; e.g. initialize a docker container then reset its database The current solution requires adding an event handler to the next lifecycle event. It is clunky because it requires the plugin writer to know the next event. In-order plugins and coordinating between tasksAs a plugin consumer, I want to use a plugin that will initialize a docker container for testing and use a second plugin to reset the container service's database to a known state before testing begins. This type of coordination isn't feasible right now because plugins (preload scripts) are loaded one after another and even if they register event handlers that return promises, all of the event handlers are called one after another making it difficult to run one task to completion before running another. It seems like this could be resolved simply with intern.registerPlugin(string: name, (options: any) => Task<void> | void): void;Registered plugins would be executed in-order and one-at-a-time. Plugin OptionsAs a plugin consumer, I want to use a plugin that will initialize a remote system and I need to supply a url to the plugin. Are plugins and tasks the same thing? Maybe we should separate the two concepts so plugins behave like preload scripts and tasks are things that plugins may register with intern and are ran in-order as described above? That would mean that plugins would register named tasks which could be ran later through configuration? If this were the case then:
Ad-hoc project codeAs a project lead, I want to write a custom code that initializes my project or testing environment. The plugin is complex enough that I need to use a module loader or load external packages (e.g. lodash). We could leverage plugins to do this, but the more we talk about it and the goals of plugins the more I think it's a distinct use case and could be addressed with a separate config parameter Setup scripts would always be loaded with the registered loader and be ran before suites are loaded. It would be nice to also provide options to the module, but I don't have a generic way to do this other than placing them on the config. Plugins making code available during testsAs a plugin writer, I want to add functionality that can be accessed from various points in intern's lifecycle. For example I want to create code that resets the database to a known state that may run as a plugin or may be called in the The current rules require that plugins are bundled, which would make it difficult to reuse code outside of when the system calls the plugin. This feels like another use case for the task pattern where a plugin registers one or more tasks that are executed automatically or can be made available later. Proposed lifecycleDepending on the outcome of the above use cases, a proposed lifecycle may look something like:
Proposed ConfigurationDepending on the outcome of the above use cases, a proposed config may look something like: {
plugins: [
'intern-docker',
'intern-mysql',
'intern-visual',
],
setup: [
'./support/custommodule.js'
],
tasks: [
{ name: 'docker', options: { action: 'run', file: './Dockerfile', ports: [3066, 3066] } },
{ name: 'mysql-reset', options: { url: 'localhost', sql: './support/test.sql' } }
]
}Looking at this config, one question I have is how would a user include a plugin so it can be found and included under a browser host or a node host? If I'm running intern in the browser then would I need to provide an absolute location or are we now assuming that all browser runs require the intern server and intern server will resolve them? |
|
Use cases! I agree, though, this is going in a good direction. Async tasksUnless we make a special plugin API with methods named "first", "second", etc., the user is going to have to have some background knowledge about when various events happen. We just need documentation that lists the events and when they might occur in relation to each other. In-order pluginsPlugins are currently loaded in-order, and any synchronous code within one is run to completion before the next one is loaded. If a config has preload: [ 'one.js', 'two.js' ]the processing order will be:
Asynchronous callbacks are not currently run in-order, they're run simultaneously(-ish). So if one.js and two.js each register a 'beforeRun' listener, the listener added by one.js will be called before the listener added by two.js, but the executor will pass the promises returned by the listener callbacks to
I agree (that was in my last post 😄). Plugin optionsI think this is the main reason for an explicit plugin API. Concerns like load ordering and async execution are already handled, but we don't currently have a good story for passing options to a plugin. What would a 'task' be? Intern does currently have some standardized components like reporters and interfaces that can be registered by name and instantiated through the config. Plugins can already interact with those simply by registering them, and it would be reasonable to do so within a plugin's initializer. Ad-hoc project codeIf you're running in Node, which is where I'd typically expect this to come up, there's nothing special to do -- For other loaders, there are a couple of options available in the existing code. One is for users to write a custom loader script that would handle setting up the loader and doing any pre-loading of modules before testing starts. Loader scripts are typically pretty short (see the examples in Another option is to do custom loading in an event callback, like: intern.on('beforeRun', () => {
return SystemJS.import('lodash').then(lodash => {
// do stuff;
});
});A new modules-to-load-with-the-loader property is certainly also an option, although I think doing that sort of thing in an event callback in a plugin (where we could pass in config data) would be more consistent and flexible. Plugins making code available during tests
I think there are two different concepts being considered here. One is for code that runs at certain points during the testing lifecycle -- that's an intern-specific plugin. The other is for code that a user might want to manually run during a test -- that a generic module. A "plugin" shouldn't need to make code available during a test, it should just perform setup actions or register event listeners. So, for example, it could listen for 'testEnd' events and clear a database after each test. Arbitrary functionality for use in tests can be provided via standard packages/modules. If a user needs to manually clear a database after each test, they could just LifecycleI think we can still accomplish all the goals mentioned above with a fairly simple lifecycle:
ConfigurationI don't think we need separate plugin and task registration. That separation is useful for things built into Intern, but if a user includes a plugin in the config, we can assume it should be loaded and run. We should use paths for the plugins rather than short names because Intern won't always be able to use node lookup semantics. In general, paths will be relative to the project root (which will typically be '/' in a browser). That's how all paths within Intern currently work, so it should feel relatively consistent to users. {
plugins: [
{
script: 'node_modules/intern-docker/index.js',
options: {
action: 'run', file: './Dockerfile', ports: [3066, 3066]
},
},
{
script: 'node_modules/intern-mysql/index.js',
options: { url: 'localhost', sql: './support/test.sql' }
}
{
script: 'node_modules/intern-visual/index.js',
options: {
...
}
},
'./support/custommodule.js'
]
} |
Async tasks
I think we can improve upon this user experience. We should be able to register a list of tasks in the configuration that happen in order. The case where a task is attached to an event handler and happens asynchronously should be the exception. Ad-hoc project codeI've been thinking about this one, and if we allow options to be passed to a task or plugin then we can write a task to address this: Plugins/Tasks can be adopted into Intern as they become standard. Tasks, Plugins, & ScriptsFrom a user experience standpoint we want to have simple stories to tell about how a user can load a plugin, write their own before and after script using their loader, and run things from a plugin. If I load a cucumber plugin, I would expect the plugin may augment Intern in some way, but I would also expect that it would provide some set of utility methods available to me during tests. As a plugin maker I have several options.
Honestly, the second option is the easiest to implement. It doesn't require a loader and if the global variable exists then I'm guaranteed that my plugin has been loaded. Otherwise, it's entirely possible for a user to use a plugin's module without the plugin having been loaded. The work-around for identifying if a plug-in has been loaded would likely follow the second pattern: create a global To avoid these issues, we should create a third option and find a place to house plug-in loaded functionality. I think allowing users to run registered tasks, or something similar in pattern is the answer. ConfigurationI'm having difficulty resolving how this configuration will be used... Questions:
I don't see how we can resolve these issues with plugins alone. I think separate task and plugin sections do work. {
plugins: [
'node_modules/intern-docker/plugin.js',
'node_modules/intern-mysql/plugin.js',
'node_modules/intern-visual/plugin.js',
'support/custommodule.js'
],
tasks: [
{ task: 'docker-run', options: { image: 'database', ports: [3066, 3066] } },
{ task: 'mysql-init', options: { url: 'localhost:3066', sql: 'support/reset.sql' } },
{ task: 'docker-run', options: { image: 'backend', ports: [80, 8888] } },
{ task: 'custom-module' }
]
}During the lifecycle plugins would be loaded as script tags and be executed immediately where they could register tasks that are executed later after intern has been initialized. Passing option data to tasks or calling them multiple times is simple because they're already registered functions. If a plugin was not loaded then it will not register its tasks and we can throw an error. We can also make tasks available through intern; i.e. |
|
I haven't wanted to derail our discussion on plugins and solving associated take-aways, but I wanted to provide a potential solution for the configuration take-away before I ran out of time. Take awaysAs a refresher, one of the take-aways we found was that configuration is hard. We wanted to have solutions for
Advanced ConfigurationOne of the issues I (and many others) have had with Intern was how the bag of options supplied in a configuration file related to what I was doing. To solve this we should consider narrowing the bag of options into separate categories related to the engine and environment selected. {
"root": {
"suites": [ "tests/unit/all" ]
},
"browserstack": {
"environments": [
{ "browser": "internet explorer", "version": [ "10", "11" ] }
],
"tunnel": "browserstack"
},
"unit": {
"engine": "node",
"extends": "root"
},
"browser": {
"engine": "browser",
"loader": "dojo",
"extends": "root"
},
"functional": {
"engine": "webdriver",
"suites": [ "tests/functional/all" ]
},
"combined": {
"runs": [ "unit", "browser", "functional" ]
},
"ci": {
"runs": [ "unit", [ "browserstack", "unit" ], [ "browserstack", "functional" ] ]
}
}The most significant change to the configuration would be the elimination of The other major change is the "runs" property. This indicates that tests are run one after another and their test coverage is combined into a single report. So in the "combined" configuation, unit tests would run in node, followed by unit tests being run in webdriver, followed by functional tests being run by webdriver. For the "ci" example, it would run "unit", then run a mixin of the browserstack and unit configuration, followed by a mixin of the browserstack and functional configuration. Regarding the cli, it would be easy to list configurations that specify an engine or add a flag to configs that should not be listed by the cli. Running a configuration in the browser (assuming the intern server is running) would look something like http://localhost/__intern?suite=browser intern would assume the configuration file would be at ./intern.json, but if we wanted to specify the config file http://localhost/__intern?suite=browser&config=intern.json or if there's only one configuration with a browser engine specified then we could automatically run it If there's multiple configuration with a ConclusionMy main goals are to simplify the configuration by moving away from the current bag of options that we have today. I think how Intern 3 forks loader options and configuration for node and browser is (using nice language) not a good pattern. I am glad to see that we are not continuing with that pattern in Intern 4's configuration. We should continue to define the specific ways in which Intern runs tests by eliminating |
|
Preliminary thoughts...
I was under the impression that we ended up with reasonable solutions for these points, at least as far as intern was concerned (vs intern-cli). So, possibly configuration is still hard. What does that actually mean?
This proposal doesn't really simplify general configuration or move away from the bag of options (it really seems to be more focused on execution rather than configuration). I mean, test writers still have to understand what options are available and how to use them to create configurations. Test runners still have to understand what executor is being run at any given time and configure their external environment accordingly (e.g., provide cloud service credentials). That said, I'm not opposed to making Intern's built-in runner more capable. One capability we're missing is a built-in way of running the Node executor and the WebDriver executor together and aggregating the results. Intern 3 provides the Combined reporter for this purpose, but the user still has to explicitly run the Node executor and the WebDriver executor. We could simplify that process by simply combining those executors, so that there would be only a single Node-based executor that would run node suites and would decide whether or not to create remote sessions based on the presence of Another feature mentioned in the proposal is the ability to list what tests will run (I'm assuming individual tests rather than just suites). I'm not really sure how valuable this feature is; it will become readily apparent which tests will run when Intern is run. However, it would be possible to do this to a limited extent by providing a |
|
Going back to plugins...
I can see when this might be useful, for consistency in test writing between Node and browsers. That could be extended to a general purpose export system for plugins, like (assume this is a built plugin): import { calculateFoo } from 'whatever';
intern.registerPlugin(options => {
intern.registerExports('plugin id', {
calculateFoo: calculateFoo
});
});A test could use the functionality something like: myTest() {
const calculateFoo = intern.getExports('plugin id').calculateFoo;
calculateFoo();
} |
|
You're right. Most of my configuration comments likely belong with I think the part I'm trying to resolve is that tests are executed in two environments: node and browser and four different modes:
These relate 1:1 to the executors except for the WebDriver executor, which does both unit and functional tests. So I don't have a clear picture of the user experience for the WebDriver case due to it's dual modes of operation. Some of the questions I have are:
From the From an Intern perspective, I'm concerned about how configuration relates to this dual mode system found in the WebDriver executor. I think we must avoid conflating node and browser runs. From a user experience perspective I would like to define 3 modes of operation: [unit] tests that run in node, functional tests that run in node driven by webdriver, and unit tests that run in the browser but can be driven by webdriver. That last definition of unit tests that run in the browser, makes it much more clear from a user experience perspective that the In order to address these issues I think we need to
If it helps, I can write up some use cases so we can be sure we're serving common patterns (or at least have plausible stories) |
|
Going back to plugins...
I like having a way of registering additional functionality that can later be called in tests. I still favor having a list of tasks that can be called via configuration. In my view: Plugins are responsible for initialization. They would register reporters, register loaders, attach event listeners, and register tasks. There would be no Tasks provide functionality that execute code and potentially register callbacks (e.g. afterRun handlers for cleanup). They would be called via configuration before tests run or explicitly during tests. This delineation makes it easy to explain what a plugin is used for (registration) and what tasks and event handlers do (execute code). |
The loader only comes into play for unit tests. WebDriver tests are fundamentally different from unit tests --- they don't load or run application code, and so have no need to use the application's loader.
Currently, plugins (well, preload scripts) will be loaded in all environments all the time, and it's up to the plugin to determine, based on the environment, whether it wants to do anything.
They (preload scripts) run now.
At the moment, plugins need to check their environment ('node', 'browser', 'webdriver') to decide whether they want to do anything. This seems like it would be desirable since it frees the user from having to worry about how to use a plugin, and the plugin writer is going to have the most knowledge about what environment the plugin was intended to run in. It would be easy enough to give the configuration more control here, though.
Ironically, that's how WebDriver behaves right now (it's only single-mode as far as things like plugins and loader are concerned), and I was planning to combine them because it should be easier for users to work with. 😄
I agree. While my initial vision was for users to write simple loader scripts in most cases, that ended up seeming like a pain.
There are two types of tests: functional and unit. Functional tests always run in Node. Unit tests may run in node or the browser. Frequently, the same set of unit tests is run in both the browser and Node environments, which is why there has traditionally only been
Having plugins register things (reporters, etc.) makes sense. Initially I was also in the "no
Hmmm...I'm still not sure what a "task" is. I mean, the concept of "plugin" is pretty well established. Plugins for tools like vim, SystemJS, Atom, or Chrome provide some inherent functionality, and may also expose functionality that can be explicitly invoked by the user. Loading a plugin will implicitly make the plugin do things, possibly controlled by the configuration passed to the plugin. For example, loading the 'vim' plugin in Atom will immediately change the behavior of the editor, as well as providing various functions that can be manually invoked by the user. Similarly installing a Chrome plugin may change the behavior of Chrome, and may also provide controls the user can interact with. Tasks sound more like the specific resources Intern supports, like reporters and interfaces; something you'd register to use later. Could you provide any specific examples of what functionality you'd like to package into a plugin and how that functionality would be broken down between "plugin" and "task"? |
Does this mean that there's no upgrade path for users with Intern 3? It sounds like we're forcing functional tests to be written with the node loader or bundle them, while Intern 3 tests were typically written as AMD modules.
If I write a plugin that works in both node and the browser. As a user (or a plugin writer) how can I ensure that the plugin only runs once when using the webdriver either before unit tests or before webdriver starts?
When you think of a task, think of grunt. Grunt has a registration phase where you register capabilities and an execution phase where you instruct grunt what tasks to run. Plugins should register capabilities and tasks are one of the ways to execute them. 99% of what a vim, SystemJS, Chrome, or Atom plugin does is attach listeners or add capabilities that can later be called (via button press or action). Even with an atom plugin that changes the behavior, look, and feel of the editor I assume it does this as part of the render callback and not immediately on the load of the plugin. We should follow this pattern. Plugins register functionality and tasks, event handlers, and intern execute it as part of its flow. I'll look into creating more user stories to help cover these use cases. |
At the moment, no, but that's a good point. So, lets say that functional tests will use whatever loader is being used for the Node environment (
Plugins will be loaded once for an executor when that executor is started. To do things at the beginning of unit tests or remote tests, a plugin should react to new sessions. The current way to do that (in Intern 3 or 4) is to listen for
That's what I'm assuming Intern plugins will do as well, hook events and potentially provide functions or resources that can be used in tests. I'm not sure how Grunt's concept of "tasks" fits in with Intern, because Intern isn't a task runner; it only has one flow of operations (load tests, run tests). Possibly we're just differing on terminology (where "task" == "exported function")? |
|
Ah, ok. I think we're basically saying the same thing.
So in your example of registering a plugin it looks basically like what I'm calling a task, but I've associated the preload script with a plugin. It's also not clear from the example how it is called later or how options are passed to it. |
|
Here's some more use cases to capture some of the scenarios we've been talking about. These are in addition to the previous set of use cases. Use CasesUpgrading from Intern 3As a user, I want to be able to upgrade directly from Intern 3 to Intern 4 with a minimal amount of effort. I would prefer only needing to write a new config.json and include a plugin. Executing Tasks multiple timesAs a user, I may want to install a plugin that provides functionality that can be executed multiple times and can be passed different options. For instance, I want to register a docker plugin and a mysql plugin to assist with testing. The docker plugin needs to start a database container, the mysql plugin will initialize the database, followed by the docker plugin starting the server container linking it to the database. I tried to come up with a real-life example, but essentially the challenge is running a task twice with different options with a task that runs in the middle of the two. I imagine the configuration may look something like this: Separate Node and Host LoadersAs a user, I want to be able to use AMD as my module loader in node (because I have a number of tests written for Intern 3), but now want to use a SystemJS loader in unit tests in the browser. I want to be able to define this in the config without needing to write a custom loader.js file. More StuffLeadFoot and DigDugI've been thinking about these two packages and wondering what makes them different from a plugin. Is there something that LeadFoot or DigDug do that we cannot reasonably do as a plugin? Are plugins missing functionality as a result? LifecycleWhat does the current, proposed WebDriver lifecycle look like? Given things we've discussed, we want to run a node lifecycle + a browser lifecycle for unit tests with each being able to configure loaders and plugins. This is what I've been putting together in my head from our discussions:
Is this essentially correct? What I'd like to see is a user able to define a remote test session that can be shared between different test runs so we can reduce the complexity of this lifecycle to two separate lifecycle one for browser with its own set of plugins and one for node (functional) with its set of plugins. We'd be able to flatten lifecycles by using two runs Browser test w/ WebDriver
(node loader or node plugin configurations are unnecessary. This lifecycle is similar to running unit tests in the browser) Functional test w/ WebDriver
(browser loader or browser plugin configuration are unnecessary because the browser is only used for functional tests) Separating the lifecycles allow for distinct configurations to be passed for browser, node, and functional tests. |
This is definitely a goal, but it hasn't yet been a priority (I've been trying to nail down the API before writing a compatibility layer/plugin).
This is starting to look like it may be outside the scope of Intern. I think it's reasonable for a plugin to be able to expose functions for use in suites and tests, but having Intern manage a bunch of generic high-level tasks like starting up docker containers seems philosophically like the kind of thing that should go in a task runner, or a custom runner script, especially when we get into things like inter-task dependencies. Maybe. I'll have to think more about that one.
This is how Intern 4 currently works.
From the individual project side, Dig Dug and Leadfoot are meant to be usable by any project, not just Intern, so they shouldn't have any explicit ties to Intern. From Intern's side, Dig Dug and Leadfoot provide core WebDriver functionality, so they're not something that a user should need to explicitly include in every project. Leadfoot is also fairly tightly bound to the WebDriver startup and management process. (A user can still use a different WebDriver in tests.) That's not to say we couldn't restructure them as plugins, although I'd be more inclined to handle them like loaders currently are rather than as general purpose plugins. That would certainly be useful if we wanted to make those components more easily swappable with something else. Intern would just need "WebDriver" and "WebDriver Tunnel" resource interfaces then plugin wrappers would need to be written for Leadfoot and Dig Dug. Doing so wouldn't affect the basic Intern 4 API, though, so we could look into this more for a future update. I'm not really sure what is meant by "Are plugins missing functionality as a result?".
It looks pretty much like what you're describing. All executors follow the same lifecycle:
Remote executors are run as suites of the local Node executor using the RemoteSuite class. For example, based on the configured environments, the local executor may end up with a RemoteSuite named "chrome - Windows 7". When this suite is run, it starts up a remote session, runs it, and collects coverage data. As the remote suite runs, its events are forwarded back to the local executor and made available for any local listeners. A Node executor will run any local suites, then run each of the remote suites. The run process looks like:
Currently a single config is shared by all executors. This is intentional, because in the past that's been the desired behavior -- the same unit tests with the same basic config are run in whatever environment you're running Intern in. If some tests were sensitive to their environment, users would use something like The current config format does allow for this use case, though, via the |
Goal is to make sure Intern 4 feels natural and efficient to use.
Ways to do this:
As you do this, you should look for things that feel suboptimal from a usability or developer ergonomics perspective, in particular:
The text was updated successfully, but these errors were encountered: