Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add smoke tests #2959

Merged
merged 1 commit into from Mar 25, 2021
Merged

add smoke tests #2959

merged 1 commit into from Mar 25, 2021

Conversation

@ruyadorno
Copy link
Contributor

@ruyadorno ruyadorno commented Mar 25, 2021

Lightweight smoke test suite that runs common commands so that we can
also have a holistic check during CI.

Overview

smoke-tests/server.js

Copy-pasted mock server from arborist

smoke-tests/content/

Package/manifest fixtures for usage along with smoke-tests/server.js

smoke-tests/index.js

Serialized, snapshot-based smoke tests that run various npm cli commands asserting the changes in state after each step.

Times

  • Average of 4s in my local, intel-based macbook pro
  • Average of 32s in GH actions ubuntu latest
  • Average of 1min in GH actions windows latest
  • Average of 50s in GH actions macos latest
@ruyadorno ruyadorno requested a review from as a code owner Mar 25, 2021
@ruyadorno ruyadorno force-pushed the add-smoke-tests branch 10 times, most recently from cead93b to 248602b Mar 25, 2021
@ruyadorno ruyadorno added this to the OSS - Sprint 27 milestone Mar 25, 2021
@ruyadorno
Copy link
Contributor Author

@ruyadorno ruyadorno commented Mar 25, 2021

NOTE: added more info in the PR description providing context on how it works.

That said, I have two items to follow up with that may or not be addressed before merging:

  • remove the [10.1, 12.1, 14.1] targets from CI (since they're likely to be very redundant in the context of smoke tests as unit tests will catch these oddities much better)
  • add smoke-tests as a step to be run during the checks phase of make publish

@ljharb
Copy link
Collaborator

@ljharb ljharb commented Mar 25, 2021

@ruyadorno as long as they’re tested then it’s fine, but smoke tests are typically slow and flaky, whereas unit tests are fast and reliable, so it seems useful to keep both.

nlf
nlf approved these changes Mar 25, 2021
Lightweight smoke test suite that runs common commands so that we can
also have a holistic check during CI.

PR-URL: npm#2959
Credit: @ruyadorno
Close: npm#2959
Reviewed-by: @nlf
@ruyadorno ruyadorno changed the base branch from latest to release-next Mar 25, 2021
@ruyadorno ruyadorno merged commit 97b4152 into npm:release-next Mar 25, 2021
@ruyadorno ruyadorno mentioned this pull request Mar 25, 2021
feelepxyz added a commit to dependabot/dependabot-core that referenced this issue Apr 16, 2021
Changes since 7.7.4: https://github.com/npm/cli/blob/latest/CHANGELOG.md

## v7.10.0 (2021-04-15)

### FEATURES

* [`f9b639eb6`](npm/cli@f9b639e)
  [#3052](npm/cli#3052)
  feat(bugs): fall back to email if provided
  ([@Yash-Singh1](https://github.com/Yash-Singh1))
* [`8c9e24778`](npm/cli@8c9e247)
  [#3055](npm/cli#3055)
  feat(version): add workspace support
  ([@wraithgar](https://github.com/wraithgar))

### DEPENDENCIES

* [`f1e6743a6`](npm/cli@f1e6743)
  `libnpmversion@1.2.0`
    * feat(retrieve-tag): retrieve unannotated git tags
    * fix(retrieve-tag): use semver to look for semver
* [`3b476a24c`](npm/cli@3b476a2)
  `@npmcl/git@2.0.8`
    * fix(git): do not use shell when calling git
* [`dfcd0c1e2`](npm/cli@dfcd0c1)
  [#3069](npm/cli#3069)
  `tap@15.0.2`

### DOCUMENTATION

* [`90b61eda9`](npm/cli@90b61ed)
  [#3053](npm/cli#3053)
  fix(contributing.md): explicitely outline dep updates
  ([@darcyclarke](https://github.com/darcyclarke))

## v7.9.0 (2021-04-08)

### FEATURES

* [`1f3e88eba`](npm/cli@1f3e88e)
  [#3032](npm/cli#3032)
  feat(dist-tag): add workspace support
  ([@nlf](https://github.com/nlf))
* [`6e31df4e7`](npm/cli@6e31df4)
  [#3033](npm/cli#3033)
  feat(pack): add workspace support
  ([@wraithgar](https://github.com/wraithgar))

### DEPENDENCIES

* [`ba4f7fea8`](npm/cli@ba4f7fe)
  `licensee@8.2.0`

## v7.8.0 (2021-04-01)

### FEATURES


* [`8bcc5d73f`](npm/cli@8bcc5d7)
  [#2972](npm/cli#2972)
  feat(workspaces): add repo and docs
  ([@wraithgar](https://github.com/wraithgar))
* [`ec520ce32`](npm/cli@ec520ce)
  [#2998](npm/cli#2998)
  feat(set-script): implement workspaces
* [`32717a60e`](npm/cli@32717a6)
  [#3001](npm/cli#3001)
  feat(view): add workspace support
  ([@wraithgar](https://github.com/wraithgar))
* [`7b177e43f`](npm/cli@7b177e4)
  [#3014](npm/cli#3014)
  feat(config): add 'envExport' flag
  ([@isaacs](https://github.com/isaacs))

### BUG FIXES

* [`4c4252348`](npm/cli@4c42523)
  [#3016](npm/cli#3016)
  fix(usage): specify the key each time for multiples
  ([@isaacs](https://github.com/isaacs))
* [`9237d375b`](npm/cli@9237d37)
  [#3013](npm/cli#3013)
  fix(docs): add workspaces configuration
  ([@wraithgar](https://github.com/wraithgar))
* [`cb6eb0d20`](npm/cli@cb6eb0d)
  [#3015](npm/cli#3015)
  fix(ERESOLVE): better errors when current is missing
  ([@isaacs](https://github.com/isaacs))

### DEPENDENCIES

* [`61da39beb`](npm/cli@61da39b)
  `@npmcli/config@2.1.0`
  * feat(config): add support for envExport:false
* [`fb095a708`](npm/cli@fb095a7)
  `@npmcli/arborist@2.3.0`:
  * [#2896](npm/cli#2896) Provide currentEdge in
  ERESOLVE if known, and address self-linking edge case.
  * Add/remove dependencies to/from workspaces when set, not root project
  * Only reify the portions of the dependency graph identified by the
  `workspace` configuration value.
  * Do not recursively `chown` the project root path.

## v7.7.6 (2021-03-29)

### BUG FIXES

* [`9dd2ed518`](npm/cli@9dd2ed5)
  fix empty newline printed to stderr
  ([@ruyadorno](https://github.com/ruyadorno))
* [`9d391462a`](npm/cli@9d39146)
  [#2973](npm/cli#2973)
  fix spelling in workspaces.md file
  ([@sethomas](https://github.com/sethomas))
* [`4b100249a`](npm/cli@4b10024)
  [#2979](npm/cli#2979)
  change 'maxsockets' default value back to 15
  ([@wallrat](https://github.com/wallrat))

### DEPENDENCIES

* [`a28f89572`](npm/cli@a28f895)
  `libnpmversion@1.1.0`
    * fix reading `script-shell` config on `npm version` lifecycle scripts
* [`03734c29e`](npm/cli@03734c2)
  `npm-packlist@2.1.5`
    * fix packaging `bundledDependencies`
* [`80ce2a019`](npm/cli@80ce2a0)
  `@npmcli/metavuln-calculator@1.1.1`
    * fix error auditing package documents with missing dependencies

## v7.7.5 (2021-03-25)

### BUG FIXES

* [`95ba87622`](npm/cli@95ba876)
  [#2949](npm/cli#2949)
  fix handling manual indexes in `npm help`
  ([@dmchurch](https://github.com/dmchurch))
* [`59cf37962`](npm/cli@59cf379)
  [#2958](npm/cli#2958)
  always set `npm.command` to canonical command name
  ([@isaacs](https://github.com/isaacs))
* [`1415b4bde`](npm/cli@1415b4b)
  [#2964](npm/cli#2964)
  fix(config): properly translate user-agent
  ([@wraithgar](https://github.com/wraithgar))
* [`59271936d`](npm/cli@5927193)
  [#2965](npm/cli#2965)
  fix(config): tie save-exact/save-prefix together
  ([@wraithgar](https://github.com/wraithgar))

### TESTS

* [`97b415287`](npm/cli@97b4152)
  [#2959](npm/cli#2959)
  add smoke tests
  ([@ruyadorno](https://github.com/ruyadorno))
jimmywarting
Copy link

jimmywarting commented on 97b4152 May 16, 2021

Maybe switch to using nodes built in mkdir instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants