test(integration): emit console error when test fails #35507

kwonoj · 2022-03-21T21:07:07Z

Minor improvements to the integration test runner to emit process's stdout when its execution failed.

We have console.log for the output, but it doesn't emit actually since jest suppresses console output unlesss assertion fails. If we throws in before* hook, since there's no actual assertion fails jest won't release captured console logs.

PR adds one straightforward workaround (https://stackoverflow.com/questions/48695717/console-log-statements-output-nothing-at-all-in-jest) to emit if process exits with non-zero errorcode. To avoid any existing behavior, this is not enabled for the cases when process successfully exits.

Bug

Related issues linked using fixes #number
Integration tests added
Errors have helpful link attached, see contributing.md

Feature

Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
Related issues linked using fixes #number
Integration tests added
Documentation added
Telemetry added. In case of a feature if it's used or not.
Errors have helpful link attached, see contributing.md

Documentation / Examples

Make sure the linting passes by running yarn lint

test/integration/production/test/index.test.js

ijjk · 2022-03-22T01:43:34Z

Failing test suites

Commit: 2c452d6

yarn testheadless test/production/emit-decorator-metadata/index.test.ts

emitDecoratorMetadata SWC option > should compile with emitDecoratorMetadata enabled
emitDecoratorMetadata SWC option > should compile with emitDecoratorMetadata enabled for API

Expand output

● emitDecoratorMetadata SWC option › should compile with emitDecoratorMetadata enabled

thrown: "Exceeded timeout of 90000 ms for a hook.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

   9 |   let next: NextInstance
  10 |
> 11 |   beforeAll(async () => {
     |   ^
  12 |     next = await createNext({
  13 |       files: {
  14 |         'jsconfig.json': new FileRef(join(__dirname, 'app/jsconfig.json')),

  at production/emit-decorator-metadata/index.test.ts:11:3
  at Object.<anonymous> (production/emit-decorator-metadata/index.test.ts:8:1)

● emitDecoratorMetadata SWC option › should compile with emitDecoratorMetadata enabled for API

thrown: "Exceeded timeout of 90000 ms for a hook.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

   9 |   let next: NextInstance
  10 |
> 11 |   beforeAll(async () => {
     |   ^
  12 |     next = await createNext({
  13 |       files: {
  14 |         'jsconfig.json': new FileRef(join(__dirname, 'app/jsconfig.json')),

  at production/emit-decorator-metadata/index.test.ts:11:3
  at Object.<anonymous> (production/emit-decorator-metadata/index.test.ts:8:1)

● Test suite failed to run

TypeError: Cannot read properties of undefined (reading 'destroy')

  24 |   })
  25 |
> 26 |   afterAll(() => next.destroy())
     |                       ^
  27 |
  28 |   it('should compile with emitDecoratorMetadata enabled', async () => {
  29 |     let browser: BrowserInterface

  at production/emit-decorator-metadata/index.test.ts:26:23

● Test suite failed to run

next instance not destroyed before exiting, make sure to call .destroy() after the tests after finished

  91 |     if (nextInstance) {
  92 |       await nextInstance.destroy()
> 93 |       throw new Error(
     |             ^
  94 |         `next instance not destroyed before exiting, make sure to call .destroy() after the tests after finished`
  95 |       )
  96 |     }

  at lib/e2e-utils.ts:93:13

Read more about building and testing Next.js in contributing.md.

This updates our default test times as discussed to account for longer `yarn install` times for the isolated tests as GitHub actions network can be slow. x-ref: #35507 (comment) x-ref: [slack thread](https://vercel.slack.com/archives/CGU8HUTUH/p1647919982840099)

ijjk · 2022-03-28T21:40:50Z

Stats from current PR

Default Build (Increase detected ⚠️)

General

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
buildDuration	14.8s	15.1s	⚠️ +283ms
buildDurationCached	6s	5.5s	-462ms
nodeModulesSize	467 MB	467 MB	✓

Page Load Tests Overall increase ✓

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.969	3.017	⚠️ +0.05
/ avg req/sec	842.14	828.64	⚠️ -13.5
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.332	1.258	-0.07
/error-in-render avg req/sec	1877.13	1986.83	+109.7

Client Bundles (main, webpack)

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
925.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42 kB	42 kB	✓
main-HASH.js gzip	28.4 kB	28.4 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	72.1 kB	72.1 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
_app-HASH.js gzip	1.36 kB	1.36 kB	✓
_error-HASH.js gzip	192 B	192 B	✓
amp-HASH.js gzip	309 B	309 B	✓
css-HASH.js gzip	327 B	327 B	✓
dynamic-HASH.js gzip	2.57 kB	2.57 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	920 B	920 B	✓
image-HASH.js gzip	5.48 kB	5.48 kB	✓
index-HASH.js gzip	263 B	263 B	✓
link-HASH.js gzip	2.26 kB	2.26 kB	✓
routerDirect..HASH.js gzip	320 B	320 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	319 B	319 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	15.2 kB	15.2 kB	✓

Client Build Manifests

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
_buildManifest.js gzip	460 B	460 B	✓
Overall change	460 B	460 B	✓

Rendered Page Sizes

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
index.html gzip	532 B	532 B	✓
link.html gzip	546 B	546 B	✓
withRouter.html gzip	526 B	526 B	✓
Overall change	1.6 kB	1.6 kB	✓

Default Build with SWC (Decrease detected ✓)

General

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
buildDuration	17.2s	17s	-200ms
buildDurationCached	5.5s	5.5s	-29ms
nodeModulesSize	467 MB	467 MB	✓

Page Load Tests Overall decrease ⚠️

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.866	2.873	⚠️ +0.01
/ avg req/sec	872.3	870.03	⚠️ -2.27
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.249	1.265	⚠️ +0.02
/error-in-render avg req/sec	2001.14	1977.03	⚠️ -24.11

Client Bundles (main, webpack)

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
925.HASH.js gzip	178 B	178 B	✓
framework-HASH.js gzip	42.3 kB	42.3 kB	✓
main-HASH.js gzip	28.8 kB	28.8 kB	✓
webpack-HASH.js gzip	1.45 kB	1.45 kB	✓
Overall change	72.7 kB	72.7 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
_app-HASH.js gzip	1.35 kB	1.35 kB	✓
_error-HASH.js gzip	179 B	179 B	✓
amp-HASH.js gzip	313 B	313 B	✓
css-HASH.js gzip	324 B	324 B	✓
dynamic-HASH.js gzip	2.56 kB	2.56 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	921 B	921 B	✓
image-HASH.js gzip	5.59 kB	5.59 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	2.33 kB	2.33 kB	✓
routerDirect..HASH.js gzip	322 B	322 B	✓
script-HASH.js gzip	388 B	388 B	✓
withRouter-HASH.js gzip	317 B	317 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	15.3 kB	15.3 kB	✓

Client Build Manifests

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
_buildManifest.js gzip	458 B	458 B	✓
Overall change	458 B	458 B	✓

Rendered Page Sizes

	vercel/next.js canary	kwonoj/next.js test-output-failed-std	Change
index.html gzip	530 B	530 B	✓
link.html gzip	544 B	544 B	✓
withRouter.html gzip	525 B	525 B	✓
Overall change	1.6 kB	1.6 kB	✓

Commit: abcff31

kwonoj requested review from timneutkens, ijjk, shuding and huozhi as code owners March 21, 2022 21:07

ijjk reviewed Mar 21, 2022

View reviewed changes

test/integration/production/test/index.test.js Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

test(integration): emit console error when test fails

7a18bdc

kwonoj force-pushed the test-output-failed-std branch from aafbcf5 to 7a18bdc Compare March 21, 2022 21:32

This comment has been minimized.

Sign in to view

Wampir0990 approved these changes Mar 22, 2022

View reviewed changes

ijjk approved these changes Mar 22, 2022

View reviewed changes

Merge branch 'canary' into test-output-failed-std

2c452d6

This comment has been minimized.

Sign in to view

ijjk mentioned this pull request Mar 28, 2022

Update default test timeouts for yarn install times #35669

Merged

Merge branch 'canary' into test-output-failed-std

abcff31

ijjk approved these changes Mar 28, 2022

View reviewed changes

kodiakhq bot merged commit abfbf41 into vercel:canary Mar 28, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 28, 2022

kwonoj deleted the test-output-failed-std branch May 9, 2022 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(integration): emit console error when test fails #35507

test(integration): emit console error when test fails #35507

kwonoj commented Mar 21, 2022

This comment has been minimized.

This comment has been minimized.

ijjk commented Mar 22, 2022

This comment has been minimized.

ijjk commented Mar 28, 2022

test(integration): emit console error when test fails #35507

test(integration): emit console error when test fails #35507

Conversation

kwonoj commented Mar 21, 2022

Bug

Feature

Documentation / Examples

This comment has been minimized.

This comment has been minimized.

ijjk commented Mar 22, 2022

Failing test suites

This comment has been minimized.

ijjk commented Mar 28, 2022

Stats from current PR