Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: --reporters flag broken #3712

Closed
3 tasks done
johnjenkins opened this issue Oct 12, 2022 · 8 comments
Closed
3 tasks done

bug: --reporters flag broken #3712

johnjenkins opened this issue Oct 12, 2022 · 8 comments
Assignees
Labels
Bug: Validated This PR or Issue is verified to be a bug within Stencil

Comments

@johnjenkins
Copy link
Contributor

johnjenkins commented Oct 12, 2022

Prerequisites

Stencil Version

~2.15

Current Behavior

When attempting to use the --reporters (https://jestjs.io/docs/cli#--reporters) flag from the cli, it throws an error [ ERROR ] runJest: TypeError: reporters.find is not a function

Expected Behavior

no error :)

Steps to Reproduce

npm init stencil

cd into the project:
npm i

add another reporter:
npm i jest-junit

try to run your tests:
npm run test -- --coverage --reporters="default" --reporters="jest-junit"

Code Reproduction URL

https://github.com/johnjenkins/stencil-reporters-flag-bug

Additional Information

It's the same issue with jest-* at v26 or v27.
If you downgrade stencil (to ~2.13 / jest-* at v26), the above steps work

@ionitron-bot ionitron-bot bot added the triage label Oct 12, 2022
@rwaskiewicz rwaskiewicz self-assigned this Oct 12, 2022
@rwaskiewicz
Copy link
Member

rwaskiewicz commented Oct 12, 2022

Hey @johnjenkins 👋

Thanks for the detailed reproduction report!

I modified your test running command ever so slightly to verify this without Jest's caching getting in the way:

- npm run test -- --coverage --reporters="default" --reporters="jest-junit"
+ npm run test -- --coverage --reporters="default" --reporters="jest-junit" --no-cache

and I believe that this was introduced somewhere in v2.17.X:

v2.17.0 (working)
$ npm i @stencil/core@2.17.0
changed 1 package, and audited 377 packages in 775ms

35 packages are looking for funding
run npm fund for details

2 high severity vulnerabilities

To address all issues (including breaking changes), run:
npm audit fix --force

Run npm audit for details.
➜ reporter-test git:(main) ✗ npm run test -- --coverage --reporters="default" --reporters="jest-junit" --no-cache

reporter-test@0.0.1 test
stencil test --spec --e2e

[15:25.1] @stencil/core
[15:25.2] v2.17.0 🚂
[15:25.3] testing e2e and spec files
[15:26.1] build, reporter-test, dev mode, started ...
[15:26.2] transpile started ...
[15:27.2] transpile finished in 1.01 s
[15:27.2] copy started ...
[15:27.2] generate custom elements started ...
[15:27.2] generate lazy started ...
[15:27.3] copy finished (0 files) in 84 ms
[15:27.4] generate custom elements finished in 171 ms
[15:27.5] generate lazy finished in 276 ms
[15:27.5] build finished in 1.39 s

[15:27.5] jest args: --coverage --reporters=default --reporters=jest-junit --no-cache --e2e --spec
--max-workers=8
PASS src/utils/utils.spec.ts
PASS src/components/my-component/my-component.spec.ts
PASS src/components/my-component/my-component.e2e.ts
-------------------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-------------------------|---------|----------|---------|---------|-------------------
All files | 100 | 100 | 100 | 100 |
components/my-component | 100 | 100 | 100 | 100 |
my-component.tsx | 100 | 100 | 100 | 100 |
utils | 100 | 100 | 100 | 100 |
utils.ts | 100 | 100 | 100 | 100 |
-------------------------|---------|----------|---------|---------|-------------------

Test Suites: 3 passed, 3 total
Tests: 8 passed, 8 total
Snapshots: 0 total
Time: 2.705 s
Ran all test suites.

From that build above, I can run cat junit.xml:

cat junit.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="8" failures="0" errors="0" time="2.547">
  <testsuite name="format" errors="0" failures="0" skipped="0" timestamp="2022-10-12T12:22:07" time="0.459" tests="4">
    <testcase classname="format returns empty string for no names defined" name="format returns empty string for no names defined" time="0.005">
    </testcase>
    <testcase classname="format formats just first names" name="format formats just first names" time="0.001">
    </testcase>
    <testcase classname="format formats first and last names" name="format formats first and last names" time="0">
    </testcase>
    <testcase classname="format formats first, middle and last names" name="format formats first, middle and last names" time="0">
    </testcase>
  </testsuite>
  <testsuite name="my-component" errors="0" failures="0" skipped="0" timestamp="2022-10-12T12:22:07" time="0.526" tests="2">
    <testcase classname="my-component renders" name="my-component renders" time="0.011">
    </testcase>
    <testcase classname="my-component renders with values" name="my-component renders with values" time="0.002">
    </testcase>
  </testsuite>
  <testsuite name="my-component" errors="0" failures="0" skipped="0" timestamp="2022-10-12T12:22:07" time="2.182" tests="2">
    <testcase classname="my-component renders" name="my-component renders" time="0.723">
    </testcase>
    <testcase classname="my-component renders changes to the name data" name="my-component renders changes to the name data" time="1.062">
    </testcase>
  </testsuite>
</testsuites>
v2.17.2 (broken)
$ npm i @stencil/core@2.17.2

changed 1 package, and audited 377 packages in 540ms

35 packages are looking for funding
run npm fund for details

2 high severity vulnerabilities

To address all issues (including breaking changes), run:
npm audit fix --force

Run npm audit for details.
➜ reporter-test git:(main) ✗ npm run test -- --coverage --reporters="default" --reporters="jest-junit" --no-cache

reporter-test@0.0.1 test
stencil test --spec --e2e

[22:52.4] @stencil/core
[22:52.5] v2.17.2 🍤
[22:52.6] testing e2e and spec files
[22:53.4] build, reporter-test, dev mode, started ...
[22:53.4] transpile started ...
[22:54.4] transpile finished in 998 ms
[22:54.4] copy started ...
[22:54.4] generate custom elements started ...
[22:54.4] generate lazy started ...
[22:54.5] copy finished (0 files) in 87 ms
[22:54.6] generate custom elements finished in 173 ms
[22:54.7] generate lazy finished in 273 ms

[ WARN ] Package Json: package.json:6:3
package.json "module" property is set to "dist/index.js". It's recommended to set the "module" property to:
./dist/components/components/index.js

  L5:  "main": "dist/index.cjs.js",
  L6:  "module": "dist/index.js",
  L7:  "es2015": "dist/esm/index.mjs",

[22:54.7] build finished in 1.35 s

[22:54.7] jest args: --no-cache --e2e --spec --coverage --reporters jest-junit --max-workers=8
Determining test suites to run...
[ ERROR ] runJest: TypeError: reporters.find is not a function

And do junit output is generated for v2.17.2

I'm going to get this ingested for us to take a closer look at.

@rwaskiewicz rwaskiewicz added the Bug: Validated This PR or Issue is verified to be a bug within Stencil label Oct 12, 2022
@ionitron-bot ionitron-bot bot removed the triage label Oct 12, 2022
@rwaskiewicz rwaskiewicz removed their assignment Oct 12, 2022
@alicewriteswrongs alicewriteswrongs self-assigned this Oct 18, 2022
@alicewriteswrongs
Copy link
Contributor

alicewriteswrongs commented Oct 27, 2022

@johnjenkins after looking into this a bit I decided that in order to address this we needed to refactor Stencil's CLI parser in order to properly support this. A PR to do so is up here: #3765

It's not yet in a prereleased, I'll ping this issue when it merges and is ready for testing!

alicewriteswrongs added a commit that referenced this issue Nov 7, 2022
This commit refactors the argument parser we use for our CLI module.
Doing so fixes an issue with certain flags supported by Jest which can
be passed multiple times. For instance, in the following example:

```
jest --coverage --reporters="default" --reporters="jest-junit"
```

all of the values for the `--reporters` flag ("default" and
"jest-junit") should be collected into an array of values, instead of
simply recording whichever value is farther to the right (Stencil's
behavior before this commit).

To support passing such arguments to the `stencil test` subcommand this
commit adds a new recursive-descent parser in `src/cli/parse-flags.ts`
to replace the somewhat ad-hoc approach that was there previously. It
parses the following grammar:

```
CLIArguments    → ""
                | CLITerm ( " " CLITerm )* ;
CLITerm         → EqualsArg
                | AliasEqualsArg
                | AliasArg
                | NegativeDashArg
                | NegativeArg
                | SimpleArg ;
EqualsArg       → "--" ArgName "=" CLIValue ;
AliasEqualsArg  → "-" AliasName "=" CLIValue ;
AliasArg        → "-" AliasName ( " " CLIValue )? ;
NegativeDashArg → "--no-" ArgName ;
NegativeArg     → "--no" ArgName ;
SimpleArg       → "--" ArgName ( " " CLIValue )? ;
ArgName         → /^[a-zA-Z-]+$/ ;
AliasName       → /^[a-z]{1}$/ ;
CLIValue        → '"' /^[a-zA-Z0-9]+$/ '"'
                | /^[a-zA-Z0-9]+$/ ;
```

The regexes are a little fuzzy, but this is sort of an informal
presentation, and additionally there are other constraints implemented
in the code which handles all of these different terms.

Refactoring this to use a proper parser (albeit a pretty simple one)
allows our implementation to much more clearly conform to this defined
grammar, and should hopefully both help us avoid other bugs in the
future and be easier to maintain.

See #3712 for more details on the issues with the `--reporters` flag in
particular.
alicewriteswrongs added a commit that referenced this issue Nov 7, 2022
This commit refactors the argument parser we use for our CLI module.
Doing so fixes an issue with certain flags supported by Jest which can
be passed multiple times. For instance, in the following example:

```
jest --coverage --reporters="default" --reporters="jest-junit"
```

all of the values for the `--reporters` flag ("default" and
"jest-junit") should be collected into an array of values, instead of
simply recording whichever value is farther to the right (Stencil's
behavior before this commit).

To support passing such arguments to the `stencil test` subcommand this
commit adds a new recursive-descent parser in `src/cli/parse-flags.ts`
to replace the somewhat ad-hoc approach that was there previously. It
parses the following grammar:

```
CLIArguments    → ""
                | CLITerm ( " " CLITerm )* ;
CLITerm         → EqualsArg
                | AliasEqualsArg
                | AliasArg
                | NegativeDashArg
                | NegativeArg
                | SimpleArg ;
EqualsArg       → "--" ArgName "=" CLIValue ;
AliasEqualsArg  → "-" AliasName "=" CLIValue ;
AliasArg        → "-" AliasName ( " " CLIValue )? ;
NegativeDashArg → "--no-" ArgName ;
NegativeArg     → "--no" ArgName ;
SimpleArg       → "--" ArgName ( " " CLIValue )? ;
ArgName         → /^[a-zA-Z-]+$/ ;
AliasName       → /^[a-z]{1}$/ ;
CLIValue        → '"' /^[a-zA-Z0-9]+$/ '"'
                | /^[a-zA-Z0-9]+$/ ;
```

The regexes are a little fuzzy, but this is sort of an informal
presentation, and additionally there are other constraints implemented
in the code which handles all of these different terms.

Refactoring this to use a proper parser (albeit a pretty simple one)
allows our implementation to much more clearly conform to this defined
grammar, and should hopefully both help us avoid other bugs in the
future and be easier to maintain.

See #3712 for more details on the issues with the `--reporters` flag in
particular.
@rwaskiewicz
Copy link
Member

rwaskiewicz commented Nov 15, 2022

Hey @johnjenkins 👋

@alicewriteswrongs' fix for this (#3765) has been included Stencil v2.19.3, which was just released this morning. I verified the fix one last time using the released version of Stencil and your reproduction case:

Original Reproduction with Stencil v2.18.1:

➜  stencil-reporters-flag-bug git:(main) npm ls @stencil/core
stencil-reporters-test@0.0.1 /sandbox/stencil-issues/3712/stencil-reporters-flag-bug
└── @stencil/core@2.18.1


➜  stencil-reporters-flag-bug git:(main)  npm run test -- --coverage --reporters="default" --reporters="jest-junit"

> stencil-reporters-test@0.0.1 test
> stencil test --spec

[09:51.8]  @stencil/core
[09:51.9]  v2.18.1 😛
[09:52.1]  testing spec files
[09:52.1]  jest args: --spec --coverage --reporters jest-junit --max-workers=8
Determining test suites to run...
[ ERROR ]  runJest: TypeError: reporters.find is not a function

Using Stencil v2.19.3

➜  stencil-reporters-flag-bug git:(main) npm i @stencil/core@latest

changed 1 package, and audited 377 packages in 723ms

35 packages are looking for funding
  run `npm fund` for details

2 high severity vulnerabilities

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
➜  stencil-reporters-flag-bug git:(main) ✗  npm run test -- --coverage --reporters="default" --reporters="jest-junit"

> stencil-reporters-test@0.0.1 test
> stencil test --spec

[10:01.1]  @stencil/core
[10:01.2]  v2.19.3 🌏
[10:01.3]  testing spec files
[10:01.3]  jest args: --spec --coverage --reporters default --reporters jest-junit --max-workers=8
 PASS  src/utils/utils.spec.ts
 PASS  src/components/my-component/my-component.spec.ts
-------------------------|---------|----------|---------|---------|-------------------
File                     | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-------------------------|---------|----------|---------|---------|-------------------
All files                |     100 |      100 |     100 |     100 |
 components/my-component |     100 |      100 |     100 |     100 |
  my-component.tsx       |     100 |      100 |     100 |     100 |
 utils                   |     100 |      100 |     100 |     100 |
  utils.ts               |     100 |      100 |     100 |     100 |
-------------------------|---------|----------|---------|---------|-------------------

=============================== Coverage summary ===============================
Statements   : 100% ( 7/7 )
Branches     : 100% ( 6/6 )
Functions    : 100% ( 3/3 )
Lines        : 100% ( 7/7 )
================================================================================

Test Suites: 2 passed, 2 of 3 total
Tests:       6 passed, 6 total
Snapshots:   0 total
Time:        0.847 s
Ran all test suites.

As a result, I'm going to close this issue. Please feel free to open a new issue if this bug resurfaces. Thanks again for the detailed bug report!

@duhem-s
Copy link

duhem-s commented Nov 16, 2022

Hello @rwaskiewicz, I still have problem with the CLI.

I used to do npm run test:unit -- mg-form to only test my mg-form component but since the version 2.19.x I'm not able to do so, all my unit tests are launched.

My test:unit script launch stencil test --spec, maybe I should do a different way ?

I also used to add the -u parameter to update my snapshots but it does not work anymore since a long time now.

@rwaskiewicz
Copy link
Member

Hey @duhem-s 👋

It sounds like there's one, potentially two different issues occurring there. For each issue, can you please open a new issue in the repo with a minimal reproduction case for the team to take a look at? Thanks!

@duhem-s
Copy link

duhem-s commented Nov 16, 2022

@rwaskiewicz Here is the issue : #3825

Should I create a new one for the -u param or can I add more info in the same issue ?

@rwaskiewicz
Copy link
Member

@duhem-s can you please create a new issue?

@duhem-s
Copy link

duhem-s commented Nov 17, 2022

@rwaskiewicz Just did it : #3828

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug: Validated This PR or Issue is verified to be a bug within Stencil
Projects
None yet
Development

No branches or pull requests

4 participants