Chutzpah.json Configuration File Examples

Chutzpah.json edit examples

Imagine you have a project with the following source and test files:

fakeSite
|   .eslintrc
|   Chutzpah.json
|
+---source
|   |   add2.js
|   |   add2.misplaced_in_source.test.js
|   |   double.js
|   |
|   \---exponential
|           square.js
|
\---tests
    |   add2.test.js
    |   double.test.js
    |
    +---exponential
    |       square.test.js
    |
    \---misplacedTests
            double.wrongFolder.test.js

If that's hard to imagine, clone the khutzpa repo and you will literally have this site on your hard drive. It's in the ./tests/fakeSite folder. The tests folder in the root of that repo.

In this testing site, we're almost following a specific convention:

For each .js source file in the folder source (and source's subfolders), have a separate file with the same name but a .test.js extension in the same folder hierarchy, using a tests folder in place of source.

For instance:

• ./source/add2.js is the source
• ./tests/add2.test.js is the test.

We have that for all of our source files, above.

But we also have some test files in the "wrong" places.

One test is in the source folder.
./source/double.wrongFolder.test.js

One test is in tests but not in the right subfolder.
./tests/misplacedTests/more.add2.test.js

More, we've got a broken test. In add2.misplaced_in_source.test.js, we test the add2broken method from add2.js. That method is broken, or at least doesn't add two to the value passed in.

---

Simplest case configuration

Let's start with a near simplest-case Chutzpah.json file for this website and see what happens.

{
    "References": [
        {
            "Path": "/source/",
            "Includes": "**/*.js",
            "Excludes": ["**/coverage/**"]
        }
    ],
    "Tests": [
        {
            "Path": "/tests/",
            "Includes": "**/*.test.js",
            "Excludes": ["**/coverage/**"]
        }
    ],
    "CodeCoverageIncludes": ["**/*.js"],
    "CodeCoverageExcludes": ["**/*.test.js"]
}

A few things to note.

1. You can use an array of strings or a single string for Includes and/or Excludes if you only have one entry.
2. Both References and Tests have Excludes of **/coverage**. That's to ignore any coverage report folders that khutzpa creates, as they will include the JavaScript files used to in the process of displaying the report, like prettify.js and sorter.js. We don't want those to factor into our tests or coverage reports.

If we use that configuration file, we'll see the following after running khutzpa:

fakeSite> khutzpa .

START:
  square.test.js
    square function in global scope
      √ should return a value that squares that which was pushed in
  double.wrongFolder.test.js
    double function in global scope
      √ should return a value double that pushed in
  add2.test.js
    add2 function in global scope
      √ should return a value that is 2 greater than that which was pushed in
      √ should return a value that is 2 when zero is sent
  add2.misplaced_in_source.test.js
    add2broken function in global scope
      × (should FAIL) should return a value that is 2 greater than that which was pushed in
  double.test.js
    double function in global scope
      √ should return the value unmodified when fed a string

Finished in 0.026 secs / 0.005 secs @ 12:26:23 GMT-0400 (Eastern Daylight Time)

SUMMARY:
√ 5 tests completed
× 1 test failed

FAILED TESTS:
  add2.misplaced_in_source.test.js
    add2broken function in global scope
      × (should FAIL) should return a value that is 2 greater than that which was pushed in
        Chrome 114.0.0.0 (Windows 10)
      Expected 27 to be 7.
          at <Jasmine>
          at UserContext.<anonymous> (source/add2.misplaced_in_source.test.js:13:28)       
          at <Jasmine>


=============================== Coverage summary ===============================
Statements   : 76.47% ( 13/17 )
Branches     : 68.75% ( 11/16 )
Functions    : 80% ( 4/5 )
Lines        : 81.25% ( 13/16 )
================================================================================
::RESULTS::
[ 1 ]
::eoRESULTS::

Note that if we check for the call's output, it's non-zero. The first non-zero return value from khutzpa's calls will be returned, here 1 taken from the single-element array, [ 1 ]. A non-zero value means failure.

To access that value on the command line, you can use the following:

Powershell

echo $LastExitCode
1

Windows Command Prompt:

echo %ERRORLEVEL%
1

---

Exclude tests by folder

Wait, so if we have a Path in Tests limiting spec files to the /tests/ folder, why is khutzpa running tests from add2.misplaced_in_source.test.js that's in /source/?

  add2.misplaced_in_source.test.js
    add2broken function in global scope
      × (should FAIL) should return a value that is 2 greater than that which was pushed in

It's because the test file got caught by our References section. When add2.misplaced_in_source.test.js is loaded, the tests will be run.

(And test files caught like this will not always run successfully, even if the tests "should" pass, depending on the order that the files are listed and loaded in References. If something the test needs is loaded later, bam, fail. Putting the file in Tests ensures it's loaded after References. Aka, don't load tests by putting them in References.)

"References": [
    {
        "Path": "/source/",
        "Includes": "**/*.js",
        "Excludes": ["**/coverage/**"]
    }
],

We can exclude add2.misplaced_in_source.test.js a few ways.

Exclude by name

"References": [
    {
        "Path": "/source/",
        "Includes": "**/*.js",
        "Excludes": ["**/add2.misplaced_in_source.test.js", "**/coverage/**"]
    }
],

Note that we've added the file name with a globstar (**) to say, "No matter where in the folder hierarchy you find this file name, exclude it." Unless there's another file by the same name that should be included, that's usually safe to do, and ensures moving a file or configuration file to another folder won't "lose" it.

Now the results won't include the misplaced tests.

START:
  square.test.js
    square function in global scope
      √ should return a value that squares that which was pushed in
  double.test.js
    double function in global scope
      √ should return the value unmodified when fed a string
  add2.test.js
    add2 function in global scope
      √ should return a value that is 2 greater than that which was pushed in  
      √ should return a value that is 2 when zero is sent
  double.wrongFolder.test.js
    double function in global scope
      √ should return a value double that pushed in

Finished in 0.025 secs / 0.004 secs @ 13:02:37 GMT-0400 (Eastern Daylight Time)

Exclude by "ends with"

If we're careful with the convention that all test files will end with *.test.js, we could simply exclude all of the files with this double-extension from the /source/, if any, to be safe.

"Excludes": ["**/*.test.js", "**/coverage/**"]

Same output as when we excluded the file by name.

Remember that single asterisks are limited to single folders and need the globstar prefix to be found anywhere other than the root. (A great primer on glob selector format is here, at node-glob.)

---

CodeCoverage

There are three properties related to code coverage in a Chutzpah.json configuration file.

1. CodeCoverageIncludes
2. CodeCoverageExcludes
3. CodeCoverageIgnores

Note that there is currently no distinction between CodeCoverageExcludes (2) and CodeCoverageIgnores (3) in khutzpa. In Chutzpah, the difference is described as follows:

CodeCoverageIgnores: This is an advanced property to use if you are having issues with CodeCoverageExcludes. When you list a file in CodeCoverageExcludes, Chutzpah will not even instrument that file using the Blanket.js engine. This can give performance improvements but in some situations can change the loading order of your files. This can cause test failures for some people. In that situation, CodeCoverageIgnores is the right option. It still lets the coverage engine instrument file but the results of coverage are ignored.

We're not using Blanket any more, and I'm honestly not 100% sure what "instrument" means in this context. Maybe we were loading the file into test runs before but now we're not?

In khutzpa, for backwards compatibility, 2 & 3 are grouped together and the sum total treated as CodeCoverageExcludes.

If you don't have a value for CodeCoverageIncludes, you'll currently still see a Coverage Summary with Unknown percentages in your results.

=============================== Coverage summary ===============================
Statements   : Unknown% ( 0/0 )
Branches     : Unknown% ( 0/0 )
Functions    : Unknown% ( 0/0 )
Lines        : Unknown% ( 0/0 )
================================================================================

These results will not prevent an otherwise successful khutzpa test run.

Together, these properties work essentially like a single entry in the References or Tests section -- you can insert multiple glob format strings in each array, but the three properties only give you one "selector" total, so to speak.

So, for us, if we want to coverage check just the items in source, we need to add...

"CodeCoverageIncludes": ["**/*.js"],

khutzpa assumes you only want coverage with items in References, and the Includes and Excludes selectors here can only produce subsets of those files.

To show how this works, let's return to the original configuration file that doesn't exclude **/*.test.js in the Excludes for References.

"References": [
    {
        "Path": "/source/",
        "Includes": "**/*.js",
        "Excludes": ["**/coverage/**"]
    }
],

Now let's run a normal run (we could also run with /coverage, but that currently only creates an html report. It doesn't output a summation to the command line).

> khutzpa .

...

=============================== Coverage summary ===============================
Statements   : 82.6% ( 19/23 )
Branches     : 68.75% ( 11/16 )
Functions    : 87.5% ( 7/8 )
Lines        : 86.36% ( 19/22 )
================================================================================

Those are pretty good numbers. Let's drill into the report to find out what it found.

File	Statements		Branches		Functions		Lines
add2.js	100%	6/6	62.5%	5/8	100%	2/2	100%	6/6
add2.misplaced_in_source.test.js	100%	6/6	100%	0/0	100%	3/3	100%	6/6
double.js	100%	3/3	100%	4/4	100%	1/1	100%	3/3

Interestingly, our .test.js file seems to be completely tested! I think that's because, during a test, every line in the test file was run. 😕🤔

But that still is unfair. We shouldn't get credit for a test running itself. Let's exclude those from our runs.

Just add this line to your file:

"CodeCoverageExcludes": ["**/*test.js"]

The coverage reported is a little lower, like we expected.

=============================== Coverage summary ===============================
Statements   : 76.47% ( 13/17 )
Branches     : 68.75% ( 11/16 )
Functions    : 80% ( 4/5 )
Lines        : 81.25% ( 13/16 )
================================================================================

Let's find out why.

File	Statements		Branches		Functions		Lines
add2.js	100%	6/6	62.5%	5/8	100%	2/2	100%	6/6
double.js	100%	3/3	100%	4/4	100%	1/1	100%	3/3

Sure enough, we lost our coverage credit for the test file, but that's only fair.

Failing tests without adequate coverage

When our tests included the file add2.misplaced_in_source.test.js we received failing return values because that file includes a broken test. When we removed that test file from our coverage report, all tests passed.

That's cool, but if the reason we don't have a failing test is because we removed a test and are no longer testing what's broken, that would be a false sense of achievement.

Luckily there's another way to measure success against a test suite: Measuring code coverage. This tells us whether we have tests for every piece of logic in our code. This is a really tedious task, and automating it makes measuring it practically possible.

Let's ensure we fail a test suite if coverage isn't high enough to be reasonably comfortable that we've tested all of our files fairly thoroughly.

We do that by adding a value for CodeCoverageSuccessPercentage. A good rule of thumb is to have 80% coverage, though this will always be a judgement call depending on the state and architecture of your code.

"CodeCoverageSuccessPercentage": 80

NOTE: The CodeCoverageSuccessPercentage property currently only works when you use the /coverage command-line property. Using CodeCoverageSuccessPercentage with the default khutzpa call will return coverage values but will not fail a run if CodeCoverageSuccessPercentage isn't met.

Let's try with a coverage percentage of 20.

"CodeCoverageSuccessPercentage": 20

The results are:

fakeSite> khutzpa . /coverage

=============================== Coverage summary ===============================
Statements   : 64.7% ( 11/17 )
Branches     : 56.25% ( 9/16 )
Functions    : 60% ( 3/5 )
Lines        : 68.75% ( 11/16 )
================================================================================
::RESULTS::
[ 0 ]        
::eoRESULTS::

That results a value of zero -- all percentages are above 20, so it passes.

Let's try 80%, however.

"CodeCoverageSuccessPercentage": 80

Now the suite fails.

fakeSite> khutzpa . /coverage

08 06 2023 23:18:06.104:ERROR [coverage]: Chrome 114.0.0.0 (Windows 10): Coverage for statements (64.7%) does not meet global threshold (80%)
08 06 2023 23:18:06.107:ERROR [coverage]: Chrome 114.0.0.0 (Windows 10): Coverage for branches (56.25%) does not 
meet global threshold (80%)
08 06 2023 23:18:06.109:ERROR [coverage]: Chrome 114.0.0.0 (Windows 10): Coverage for lines (68.75%) does not meet global threshold (80%)
08 06 2023 23:18:06.110:ERROR [coverage]: Chrome 114.0.0.0 (Windows 10): Coverage for functions (60%) does not meet global threshold (80%)

=============================== Coverage summary ===============================
Statements   : 64.7% ( 11/17 )
Branches     : 56.25% ( 9/16 )
Functions    : 60% ( 3/5 )
Lines        : 68.75% ( 11/16 )
================================================================================
::RESULTS::
[ 1 ]
::eoRESULTS::

And there we have it... with /coverage and a CodeCoverageSuccessPercentage higher than at least one of the values in the summary, we have a 1 returned, and the suite fails.