Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Intern's configuration system #1109

Closed
jason0x43 opened this issue Feb 21, 2020 · 0 comments · Fixed by #1149
Closed

Update Intern's configuration system #1109

jason0x43 opened this issue Feb 21, 2020 · 0 comments · Fixed by #1149
Assignees
Labels
effort-high This will take a while enhancement A new or improved feature priority-high Most important

Comments

@jason0x43
Copy link
Member

jason0x43 commented Feb 21, 2020

Intern's usage over the last couple of years has exposed some areas for improvement in Intern's configuration system.

Problems

The main problems are:

  1. The config-processing logic is spread out and can be hard to follow
  2. The merging logic is inconsistent (Additive configuration is broken in some instances. #877)
  3. The self tests aren't fully exercising the merge logic
  4. Some of the default options aren't applied consistently, leading to issues like Additive configuration is broken in some instances. #877, where attempting to add a reporter to the default simply replaces the default.

Background

The current process is split between core/lib/<env>/util#getConfig, core/common/util#loadConfig, and core/lib/executors/<executor>#_resolveConfig.

  1. Load a config file (the default or one given as a command line arg or query arg) - getConfig
  2. Process the config file, normalizing config values and processing merge directives - loadConfig
  3. Apply command line or query arguments - loadConfig
  4. "Resolve" the config (expand globs) - _resolveConfig. During this process, defaults stored in the executors are applied.

The config file format is JSON+comments.

The config file format is intentionally a bit looser than the internal config format for ease of use. For example, a config file can use a string for the suites property, but in a Config structure the suites property is always an array of strings.

Various parts of the config file may be merged during processing:

  • A config that uses extends will be merged with whatever config it's extending
  • Child configs (children of a configs property) are merged with the parent config, or with the config they explicitly extend
  • Resource properties (suites, plugins) are implicitly added to resource properties in node and browser properties
  • A few properties (suites, plugins, instrumenterOptions, tunnelOptions) can be suffixed with a '+', in which case they'll be shallowly mixed into the current value of the property at the time they're processed

One substitution is currently performed: The string {pwd} in environment descriptors will be expanded to the current working directory at runtime.

Desired outcomes

Externalize the defaults

Apply the default config options currently setup in the executor constructors in the main config handling logic. The order should be:

  1. Default config
  2. Loaded config
  3. Environment variables (INTERN_ARGS)
  4. CLI / query args

This will require moving the default options into the config logic, or into separate environment-specific files.

Merging should always be explicit.

Given this:

{
  "suites": [ "a.ts", "b.ts" ],
  "node": { "suites": [ "c.ts" ] }
}

only c.ts should be used in a node environment. To get the existing behavior, use:

{
  "suite": [ "a.ts", "b.ts" ],
  "node": { "suites+": [ "c.ts" ] }
}

Merging should always be shallow

Merging should be shallow (it is now). Users can use '+' suffixes to explicitly merge subproperties.

Improved test coverage

  • Verify that default capabilities are mixed into config
  • Verify that '+' directives are merged properly for extended configs and from child configs.
@jason0x43 jason0x43 added enhancement A new or improved feature effort-high This will take a while priority-high Most important labels Feb 21, 2020
@jason0x43 jason0x43 self-assigned this Apr 16, 2020
jason0x43 added a commit that referenced this issue Apr 17, 2020
Consolidate common config code into core/lib/config/*

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue Apr 20, 2020
jason0x43 added a commit that referenced this issue Apr 20, 2020
- Remove 'Config' generic from executors
- Improve parseValue types
- Consolidate properties in mixinProperty

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue Apr 22, 2020
- Consolidate more config code into lib/config
- Set all general config defaults in Executor
- Move static values out of _resolveConfig

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue Apr 24, 2020
- Improve separation of config operations in Configurator
- Better names for key config API functions
- More config error classes
- use createConfigurator factory instead of Configurator constructor

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue Apr 30, 2020
- Finish updating the config processing code
- Update commander

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue Apr 30, 2020
This is still a WIP

- Improve config types
- All unused destructure properties (eslint)
- Simplify config process
- Fix some broken unit tests

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue May 1, 2020
- Node executor tests pass
- Add some tests for core/lib/config

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue May 4, 2020
This is now handled by the UI (cli/index)

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue May 4, 2020
jason0x43 added a commit that referenced this issue May 4, 2020
jason0x43 added a commit that referenced this issue May 5, 2020
- Reorganize lib/config -- move individual modules under a lib/
  directory
- Extract some Selenium typings to a standalone tunnels/types module to
  allow them to be used in config code

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue May 5, 2020
- Reorganize lib/config -- move individual modules under a lib/
  directory
- Extract some Selenium typings to a standalone tunnels/types module to
  allow them to be used in config code

references #1109

[ci skip]
jason0x43 added a commit that referenced this issue May 13, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged.
jason0x43 added a commit that referenced this issue May 14, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged.
jason0x43 added a commit that referenced this issue May 14, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For exmaple, `capabilities` in a config file will no
longer be implicitly merged with the default. Use `capabilities+` to
retain the default.
jason0x43 added a commit that referenced this issue May 15, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For exmaple, `capabilities` in a config file will no
longer be implicitly merged with the default. Use `capabilities+` to
retain the default.
jason0x43 added a commit that referenced this issue May 15, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For exmaple, `capabilities` in a config file will no
longer be implicitly merged with the default. Use `capabilities+` to
retain the default.
jason0x43 added a commit that referenced this issue May 17, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executors no longer have a Config-based generic
parameter.
BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For exmaple, `capabilities` in a config file will no
longer be implicitly merged with the default. Use `capabilities+` to
retain the default.
jason0x43 added a commit that referenced this issue May 18, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For example, `capabilities` in a config file will no
longer be implicitly merged with the default (use `capabilities+` to
retain the default).
jason0x43 added a commit that referenced this issue May 18, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For example, `capabilities` in a config file will no
longer be implicitly merged with the default (use `capabilities+` to
retain the default).
jason0x43 added a commit that referenced this issue Jul 21, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Move mockImport and rewiremock into devPackages/test-util
- Improve Node executor's handling of non-sibling packages by resolving
  scripts against config.basePath
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For example, `capabilities` in a config file will no
longer be implicitly merged with the default (use `capabilities+` to
retain the default).
jason0x43 added a commit that referenced this issue Aug 10, 2020
- Consolidate config logic into src/core/lib/config
- Move side effects and non-configuration code out of _resolveConfig in
  executors
- Break config functionality down into simpler, composable parts
- Add `Configurator` class -- this class encapsulates an
  environment-specific (Node or browser) config handler
- Add new error classes for config loading errors
- Remove the Config-based generic parameter from Executor
- Deprecate `help` config property
- Remove `showConfig` handling from Executor and deprecate the `showConfig`
  config property. This is now handled by the UI.
- Add a `describe` command to the CLI and deprecate the `showConfigs` config
  property
- All config defaults are now set in the Executor constructor
- Move Selenium typings to a standalone module so they can be used in
  common config code
- Don't duplicate _rootSuite setup in Node executor
- Update config unit tests
  - Test updated API
  - Extensive typing updates to config and tunnel tests
  - Add tests to more fully exercise config logic, including various
    merging behaviors
  - Replace mock classes in config and Node executor tests with real classes
    using mocked dependencies
  - Add new deepEqualExcludes chai assertion
- Move mockImport and rewiremock into devPackages/test-util
- Improve Node executor's handling of non-sibling packages by resolving
  scripts against config.basePath
- Update config docs
- Update commander dependency

references #1109

BREAKING CHANGE: Executor no longer implicitly handles the full config
process (command line args, env vars, etc.). The user will need to
manage more of Intern's configuration when using it programmatically.
BREAKING CHANGE: No config properties, aside from `configs`, are
implicitly merged. For example, `capabilities` in a config file will no
longer be implicitly merged with the default (use `capabilities+` to
retain the default).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort-high This will take a while enhancement A new or improved feature priority-high Most important
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant