Auto-generated unit tests from JSDoc

[gziolo]

Description

We are using JSDoc's @example token as of today to include code examples with the usage of public API methods. The issue is that those examples can get out of date and it won't get noticed until someone reports it. I'd like us to seek to improve this workflow and introduce internal auto-generated unit tests for those examples which could also play a role of better documentation at the same time.

Example:

/**
 * Count some words.
 *
 * @param {String} text         The text being processed
 * @param {String} type         The type of count. Accepts ;words', 'characters_excluding_spaces', or 'characters_including_spaces'.
 * @param {Object} userSettings Custom settings object.
 *
 * @example
 * ```js
 * import { count } from '@wordpress/wordcount';
 * const numberOfWords = count( 'Words to count', 'words', {} )
 * ```
 *
 * @return {Number} The word or character count.
 */
export function count( text, type, userSettings ) {
	if ( '' === text ) {
		return 0;
	}

	if ( text ) {
		const settings = loadSettings( type, userSettings );
		const matchRegExp = settings[ type + 'RegExp' ];
		const results = ( 'words' === settings.type ) ?
			matchWords( text, matchRegExp, settings ) :
			matchCharacters( text, matchRegExp, settings );

		return results ? results.length : 0;
	}
}

Proposal

Update: Edited based on feedback from @aduth in #15735 (comment).

Add handling for the result of operation assigned to result constant in @example JSDoc token:

/**
 * @example
 * ```js
 * import { count } from '@wordpress/wordcount';
 * count( 'Words to count', 'words', {} );
 * // => 3
 * ```
 */

Behind the scenes, we would generate test files in a similar way we do it for documentation. In the case of @wordpress/wordcount package we could create packages/wordcount/src/test/public-api.jsdoc.js file, hoist and deduplicates all imports (we should allow only imports from WordPress packages for simplicity) and wrap everything else with description and test. In addition, the code comment with return value would be used to create assertion matcher.

Example:

// `public-api.jsdoc.js` - this file is auto-generated with `npm run docs:build`.
import { count } from '@wordpress/wordcount';

description( 'wordcount - public API', () => {
    test( 'count', () => {
        const result = count( 'Words to count', 'words', {} );
        expect( result ).toBe( 3 );
    } );
} );

[youknowriad]

See this for inspiration https://github.com/hoaproject/Kitab (cc @Hywan)

[gziolo]

@youknowriad shared also this npm package in our private discussion:
https://yamadapc.github.io/jsdoctest
It integrates with Mocha but provides the same functionality I was thinking about.

[oandregal]

Also for inspiration: python has something like this.

[aduth]

This would be great.

A few questions on specifics:

• Would these expected results be included anywhere in the documentation? I actually quite like the jsdoctest example where the conventional // => expected result could be used both for documentation and for constructing a test case. At the very least, from the provided proposal, the expected 3 value would be nice to include somewhere in the documented example.
• I wonder if we still might run the code for examples without the expected result, to at least check against runtime errors (which would be quite likely in cases where code has become outdated).

[gziolo]

• Would these expected results be included anywhere in the documentation? I actually quite like the jsdoctest example where the conventional // => expected result could be used both for documentation and for constructing a test case. At the very least, from the provided proposal, the expected 3 value would be nice to include somewhere in the documented example.

This seems like a great proposal as we could operate using @example exclusively 👍
I will update the description according to your proposal.

• I wonder if we still might run the code for examples without the expected result, to at least check against runtime errors (which would be quite likely in cases where code has become outdated).

Yes, we totally could do that as well 👍

[gziolo]

Related #18031. StoryShots is very similar but for stories written for Storybook. This will solve the issue for UI components.

[sainthkh]

It should be tackled after #21238 is merged.

[gziolo]

It should be tackled after #21238 is merged.

Awesome, @sainthkh let me know when you have PR opened or is it that you wanted to highlight the dependency? 😄

[sainthkh]

Actually both. I wanted to highlight the dependency to make sure some volunteers mistakenly try to implement this before #21238 is done.

And I also want to tackle it myself, too.

[aduth]

Even prior to implementing any automation, it seems like a related task would be to ensure there's consistency in the existing documentation for how these "expected results" values are notated. My earlier comment #15735 (comment) mentioned the // => convention. If we want to settle on this, we can already start to update existing code samples today, which would make future integration more seamless, and also bring some consistency to auto-documentation.