-
Notifications
You must be signed in to change notification settings - Fork 29.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_runner: add snapshot testing #53169
Conversation
Review requested:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, good tests, left some comments, want to see how people feel about this but personally I'm +1
I believe snapshot does not belong in core because it's too specific to a domain. |
exports[`suite of snapshot tests > snapshot test 1`] = ` | ||
{ | ||
"value1": 1, | ||
"value2": 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cjihrig It seems like this is missing backtick consistency for string values in snapshots.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what you mean.
I think this is a little too vague. What domain? What can't we do that most other test runners have figured out how to do? |
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) #53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) #53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) #53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379
@cjihrig is there any reason JSON.stringify was chosen here over Seems like it'd be a good fit here? |
Yes. Per the
It's not a suitable default for Node core, but you can use it for your own serialization if you want. |
Ah, lame. Alright, thanks for the heads up! It'd be nice if the stance was changed on that api to only change output across majors. It seems reasonable to me that you might have to update your snapshots when upgrading to a new major. Oh well! |
I would also love to see someone get a better and reliable serialization format in place! |
util.inspect was actually used in the past iteration of snapshot testing we had and it caused problems people complained about (about how it defines equality and semverness) |
This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: nodejs#53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: nodejs#53169 Refs: nodejs#52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This commit adds a t.assert.snapshot() method that implements snapshot testing. Serialization uses JSON.stringify() by default, but users can configure the serialization to meet their needs. PR-URL: nodejs#53169 Fixes: nodejs#48260 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) nodejs#53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) nodejs#53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) nodejs#53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) nodejs#52762 * add StefanStojanovic to collaborators (StefanStojanovic) nodejs#53118 * add Marco Ippolito to TSC (Rafael Gonzaga) nodejs#53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) nodejs#53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) nodejs#51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) nodejs#52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) nodejs#52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) nodejs#53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) nodejs#52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) nodejs#53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) nodejs#53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) nodejs#52848 PR-URL: nodejs#53379
This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: nodejs#53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: nodejs#53169 Refs: nodejs#52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This commit adds a t.assert.snapshot() method that implements snapshot testing. Serialization uses JSON.stringify() by default, but users can configure the serialization to meet their needs. PR-URL: nodejs#53169 Fixes: nodejs#48260 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) nodejs#53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) nodejs#53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) nodejs#53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) nodejs#52762 * add StefanStojanovic to collaborators (StefanStojanovic) nodejs#53118 * add Marco Ippolito to TSC (Rafael Gonzaga) nodejs#53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) nodejs#53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) nodejs#51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) nodejs#52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) nodejs#52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) nodejs#53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) nodejs#52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) nodejs#53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) nodejs#53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) nodejs#52848 PR-URL: nodejs#53379
Looking at that code, this method appears duplicated: node/lib/internal/test_runner/test.js Lines 1236 to 1244 in e08a654
node/lib/internal/test_runner/test.js Lines 228 to 230 in e08a654
Could be simplified with node/lib/internal/test_runner/test.js Lines 546 to 550 in e08a654
I.e. the test could have just had a |
This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: #53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: #53169 Refs: #52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: #53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>
When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: #53169 Refs: #52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: TODO
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53945
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53945
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53945
Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53945
test_runner: add context.fullName
This commit adds a fullName getter to the TestContext and
SuiteContext classes. This is similar to the existing name getter,
but also includes the name of all ancestor tests/suites.
doc: add context.assert docs
When context.assert was added, no docs were added. This commit
adds initial documentation for context.assert because the
snapshot() function requires them.
Refs: #52860
test_runner: add snapshot testing
This commit adds a t.assert.snapshot() method that implements
snapshot testing. Serialization uses JSON.stringify() by default,
but users can configure the serialization to meet their needs.
Fixes: #48260