New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assert: add error diffs #17615

Closed
wants to merge 5 commits into
base: master
from

Conversation

@BridgeAR
Member

BridgeAR commented Dec 11, 2017

The current assert errors are actually pretty bad when it comes to objects. This implements
a simple way to add diffs as default but only in assert strict mode. It relies on another PR that changes the util.inspect output as I use that for the diff. Building the diff differently would likely yield better results but it would also mean more work and this on its own should already be a significant improvement.

So far a few spots are not optimized for performance and I would like to get some general feedback on the solution. I especially do not like the way to distinguish asserts strict mode and legacy mode.
Activating the diff always would mean we have to change all our assert tests and I did not want to do that, besides the fact that some people might partially rely on the message output.

Refs #17576

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

assert, lib, util, errors

@@ -26,12 +29,33 @@ When using the `strict mode`, any `assert` function will use the equality used i
the strict function mode. So [`assert.deepEqual()`][] will, for example, work the
same as [`assert.deepStrictEqual()`][].
Besides that error messages that involve objects produce a error diff instead of

This comment has been minimized.

@Trott

Trott Dec 12, 2017

Member

Nit: a errror diff -> an error diff

The sentence seems to be missing some punctuation for clarity. Probably a comma after the second word?

@Trott Trott added the in progress label Dec 12, 2017

It can be accessed using:
```js
const assert = require('assert').strict;
```
Example error diff:
```diff

This comment has been minimized.

@vsemozhetbyt

vsemozhetbyt Dec 12, 2017

Member
  1. I am not sure, but maybe it is worth to separate the input with js label (for better highlighting and linting) and the output with diff label.

  2. I cannot find any ```diff insets in .md files in this repo or nodejs.org repo. Are we sure they are rendered to the .html docs properly?

@BridgeAR BridgeAR removed the in progress label Dec 28, 2017

@BridgeAR BridgeAR changed the title from [WIP] assert: add error diffs to assert: add error diffs Dec 28, 2017

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Dec 28, 2017

I just updated the code. I also improved the diffing algorithm from the first draft. It will now try to optimize the output rudimentary by removing identical lines from the back first, limits the output lines and some other smaller changes.

So PTAL

@jasnell

This comment has been minimized.

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Dec 29, 2017

Rebased due to conflicts.

I actually changed the tests to use the new assert.throws(fn, obj) pattern instead of try {} catch {} or common.expectsError.

@BridgeAR

This comment has been minimized.

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 3, 2018

@nodejs/collaborators PTAL. This should be ready but it would be nice to get another LG.

@TimothyGu TimothyGu added author ready and removed author ready labels Jan 13, 2018

const expectedLines = util
.inspect(expected, { compact: false, depth: 10 }).split('\n');
const msg = `Input A expected to ${operator} input B:\n` +
'\u001b[32m+ expected\u001b[39m \u001b[31m- actual\u001b[39m';

This comment has been minimized.

@mcollina

mcollina Jan 15, 2018

Member

We should probably be testing that unix color codes are supported on the current terminal.

This comment has been minimized.

@addaleax

addaleax Jan 15, 2018

Member

I don’t think we can test that but it might make sense to:

  • Check whether process.stderr is a TTY
  • Make coloring overridable in some way

This comment has been minimized.

@BridgeAR

BridgeAR Jan 15, 2018

Member

That would indeed be nice... I will see what I can do but I am not sure if we can properly detect that.

@@ -26,12 +29,42 @@ When using the `strict mode`, any `assert` function will use the equality used i
the strict function mode. So [`assert.deepEqual()`][] will, for example, work the
same as [`assert.deepStrictEqual()`][].
Besides that error messages that involve objects produce an error diff instead of

This comment has been minimized.

@addaleax

addaleax Jan 15, 2018

Member

grammar nit: comma after the first that? Or maybe just start with Error messages which involve objects …?

@@ -152,7 +152,8 @@ assert.deepStrictEqual = function deepStrictEqual(actual, expected, message) {
expected,
message,
operator: 'deepStrictEqual',
stackStartFn: deepStrictEqual
stackStartFn: deepStrictEqual,
errorDiff: this === strict ? 2 : 0

This comment has been minimized.

@addaleax

addaleax Jan 15, 2018

Member

Can you name the different error diff modes? It’s really hard to keep track of constants 0, 1 and 2 with distinct meanings, especially in code that spans across files.

const expectedLines = util
.inspect(expected, { compact: false, depth: 10 }).split('\n');
const msg = `Input A expected to ${operator} input B:\n` +
'\u001b[32m+ expected\u001b[39m \u001b[31m- actual\u001b[39m';

This comment has been minimized.

@addaleax

addaleax Jan 15, 2018

Member

I don’t think we can test that but it might make sense to:

  • Check whether process.stderr is a TTY
  • Make coloring overridable in some way
@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 16, 2018

@mcollina @addaleax I added a util.hasColors function. It is a very basic way to detect this but it seems like it is better than anything we had so far.

As this is probably also something that other users might want to rely on, I added it to the util module, even though I am not sure if it is the best place for it. What do you think about that?
And about adding it as a dedicated function in general: if we expose this, other users will report things to improve the current implementation :-).

@jasnell @Trott @vsemozhetbyt @TimothyGu please also take another look.

added: REPLACEME
-->
* `stream` {Stream} A [stream][]. Defaults to `process.stdout`.

This comment has been minimized.

@vsemozhetbyt

vsemozhetbyt Jan 16, 2018

Member

[stream][] — Missing bottom reference?

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 16, 2018

Rebased and new CI https://ci.nodejs.org/job/node-test-pull-request/12554/

I also addressed the doc comment.

@mcollina

LGTM

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 16, 2018

Thanks for the reviews. This is going to be tedious though because lots of systems are failing because they all use different envs settings and support different things.

So I try to fix that now.

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 16, 2018

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Jan 16, 2018

I decided to remove the part that breaks a test and to make that a separate PR.

New CI https://ci.nodejs.org/job/node-test-pull-request/12561/

]
```
If you want to deactivate the colors, use the `NODE_DISABLE_COLORS` environment

This comment has been minimized.

@jasnell

jasnell Jan 16, 2018

Member

nit: avoid using you in the docs

BridgeAR added a commit to BridgeAR/node that referenced this pull request Mar 8, 2018

tty: add getColorDepth function
Right now it is very difficult to determine if a terminal supports
colors or not. This function adds this functionality by detecting
environment variables and checking process.

PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

BridgeAR added a commit to BridgeAR/node that referenced this pull request Mar 8, 2018

tty: refactor to es6
PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

@BridgeAR BridgeAR referenced this pull request Mar 8, 2018

Closed

[9.x backport] assert & util PRs #19230

4 of 4 tasks complete
@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Mar 8, 2018

Backport opened in #19230

MylesBorins added a commit that referenced this pull request Mar 15, 2018

assert: improve error messages
From now on all error messages produced by `assert` in strict mode
will produce a error diff.

Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MylesBorins added a commit that referenced this pull request Mar 15, 2018

tty: add getColorDepth function
Right now it is very difficult to determine if a terminal supports
colors or not. This function adds this functionality by detecting
environment variables and checking process.

Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MylesBorins added a commit that referenced this pull request Mar 15, 2018

tty: refactor to es6
Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

@targos targos referenced this pull request Mar 18, 2018

Merged

v9.9.0 proposal #19428

MylesBorins added a commit that referenced this pull request Mar 20, 2018

assert: improve error messages
From now on all error messages produced by `assert` in strict mode
will produce a error diff.

Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MylesBorins added a commit that referenced this pull request Mar 20, 2018

tty: add getColorDepth function
Right now it is very difficult to determine if a terminal supports
colors or not. This function adds this functionality by detecting
environment variables and checking process.

Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MylesBorins added a commit that referenced this pull request Mar 20, 2018

tty: refactor to es6
Backport-PR-URL: #19230
PR-URL: #17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MylesBorins added a commit that referenced this pull request Mar 21, 2018

2018-03-21, Version 9.9.0 (Current)
Notable changes:

* assert:
  - From now on all error messages produced by `assert` in strict mode
    will produce a error diff. (Ruben Bridgewater)
    #17615
  - From now on it is possible to use a validation object in throws
    instead of the other possibilities. (Ruben Bridgewater)
    #17584
* crypto:
  - allow passing null as IV unless required (Tobias Nießen)
    #18644
* fs:
  - support as and as+ flags in stringToFlags() (Sarat Addepalli)
    #18801
* tls:
  - expose Finished messages in TLSSocket (Anton Salikhmetov)
    #19102
* tty:
  - Add getColorDepth function to determine if terminal supports colors
    (Ruben Bridgewater) #17615
* util:
  - add util.inspect compact option (Ruben Bridgewater)
    #17576
* **Added new collaborators**
  - [watson](https://github.com/watson) Thomas Watson

PR-URL: #19428

MylesBorins added a commit that referenced this pull request Mar 21, 2018

2018-03-21, Version 9.9.0 (Current)
Notable changes:

* assert:
  - From now on all error messages produced by `assert` in strict mode
    will produce a error diff. (Ruben Bridgewater)
    #17615
  - From now on it is possible to use a validation object in throws
    instead of the other possibilities. (Ruben Bridgewater)
    #17584
* crypto:
  - allow passing null as IV unless required (Tobias Nießen)
    #18644
* fs:
  - support as and as+ flags in stringToFlags() (Sarat Addepalli)
    #18801
* tls:
  - expose Finished messages in TLSSocket (Anton Salikhmetov)
    #19102
* tty:
  - Add getColorDepth function to determine if terminal supports colors
    (Ruben Bridgewater) #17615
* util:
  - add util.inspect compact option (Ruben Bridgewater)
    #17576
* **Added new collaborators**
  - [watson](https://github.com/watson) Thomas Watson

PR-URL: #19428

FallenRiteMonk added a commit to FallenRiteMonk/node that referenced this pull request Mar 23, 2018

2018-03-21, Version 9.9.0 (Current)
Notable changes:

* assert:
  - From now on all error messages produced by `assert` in strict mode
    will produce a error diff. (Ruben Bridgewater)
    nodejs#17615
  - From now on it is possible to use a validation object in throws
    instead of the other possibilities. (Ruben Bridgewater)
    nodejs#17584
* crypto:
  - allow passing null as IV unless required (Tobias Nießen)
    nodejs#18644
* fs:
  - support as and as+ flags in stringToFlags() (Sarat Addepalli)
    nodejs#18801
* tls:
  - expose Finished messages in TLSSocket (Anton Salikhmetov)
    nodejs#19102
* tty:
  - Add getColorDepth function to determine if terminal supports colors
    (Ruben Bridgewater) nodejs#17615
* util:
  - add util.inspect compact option (Ruben Bridgewater)
    nodejs#17576
* **Added new collaborators**
  - [watson](https://github.com/watson) Thomas Watson

PR-URL: nodejs#19428
return COLORS_16;
return COLORS_2;
};

This comment has been minimized.

@sindresorhus

sindresorhus Mar 23, 2018

This code is mostly copy-pasted from https://github.com/chalk/supports-color/blob/master/index.js and https://github.com/isaacs/color-support/blob/master/index.js with minor style tweaks. I don't see any attribution.

This comment has been minimized.

@richardlau

richardlau Mar 23, 2018

Member

@BridgeAR can you comment on the above?

This comment has been minimized.

@BridgeAR

BridgeAR Mar 24, 2018

Member

Hey, I just came home and saw this, so I could not respond any earlier.

When writing on this I got the #17615 (comment) from @mcollina that I should check for the colors. So I googled a lot about checking color support in terminals and I tried to gather as much information as I could (I actually tried hard not to rely on environment variables at all). In the end I got inspired by multiple sources including chalk, color-support, different gists, stackoverflow and more that I do not remember. I did not think of anything bad when adding this function. I do see the similarities though and I am more than happy to give the attributions. I am just not sure how to do this properly right now as there are multiple licenses.

@sindresorhus shall I just move the function in a new file and add both license texts? I am not sure about the other sources though. Or would it be fine to add something like: This functionality got inspired by multiple sources including 'chalk' by @sindresorhus and 'color-support' by @isaacs?

This comment has been minimized.

@BridgeAR

This comment has been minimized.

@richardlau

richardlau Mar 24, 2018

Member

I guess also cc @nodejs/tsc FYI

This comment has been minimized.

@joyeecheung

joyeecheung Mar 28, 2018

Member

@BridgeAR
I think moving the color detection code to an internal lib file and put attributions there would be better than doing it in tty.js. But it still comes down to whether @sindresorhus and @isaacs accept it or not.

@addaleax addaleax referenced this pull request Mar 28, 2018

Closed

console: add color support #19372

4 of 4 tasks complete

@targos targos referenced this pull request Apr 2, 2018

Closed

test: improve tty.getColorDepth coverage #19446

4 of 4 tasks complete

BridgeAR added a commit to BridgeAR/node that referenced this pull request May 1, 2018

test: actually test tty `getColorDepth()`
TTY tests should almost never be placed in `/parallel/`. Skipping TTY
tests there due to missing tty fds just means they will never be run,
ever, on any system.

This moves the tty-get-color-depth test to `/pseudo-tty/` where the test
runner will actually make a pty fd.

Refs: nodejs#17615
PR-URL: nodejs#18800
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

assert: improve error messages
From now on all error messages produced by `assert` in strict mode
will produce a error diff.

PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

lib: add internal removeColors helper
Instead of having three times the same RegExp, just use a helper.

PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

tty: add getColorDepth function
Right now it is very difficult to determine if a terminal supports
colors or not. This function adds this functionality by detecting
environment variables and checking process.

PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

tty: refactor to es6
PR-URL: nodejs#17615
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

test: actually test tty `getColorDepth()`
TTY tests should almost never be placed in `/parallel/`. Skipping TTY
tests there due to missing tty fds just means they will never be run,
ever, on any system.

This moves the tty-get-color-depth test to `/pseudo-tty/` where the test
runner will actually make a pty fd.

Refs: nodejs#17615
PR-URL: nodejs#18800
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
@MylesBorins

This comment has been minimized.

Member

MylesBorins commented Jul 31, 2018

Requested backport to 8.x in #19230

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment