compatibility-kit: [Doc] Usage of CCK and fake-cucumber

[bbros-dev]

Summary

It is necessary to use the Cucumber-Compatibility-Kit (CCK) to track, and document, the status of a new Cucumber/Gherkin implementation, or to update the status of an existing Cucumber implementation.

However, the documentation is cryptic, and there is no working example(s) of building a CCK report. Naming the golden-source/definitive source fake-cucumber did not reduce confusion.

Expected Behavior

Documentation that when followed leaves:

• Updated testdata generated by golden-master (fake-cucumber?)
• Updated CCK report and artifacts.

Current Behavior

No effective documentation.

Possible Solution

To update the status of Cucumber-<Lang> compared to golden-master (fake-cucumber):

• Requirements:
  • Install node.js >15.0 or from NodeSource
  • Install npm
  • git clone https://github.com/cucumber/cucumber ~/src/cucumber
  • git clone https://github.com/bbqsrc/gherkin-rust ~/src/cucumber-rust

pushd ~/src/cucumber/fake-cucumber
  npm install -g fake-cucumber
popd
pushd ~/src/cucumber/compatibility-kit
  make clean
  make
popd

At this point we get a multitude of errors:

error TS2307: Cannot find module '@cucumber/fake-cucumber' or its corresponding type declarations.

We aren't node gurus - and don't want to be - so we hacked around this trying to install fake-cucmber various ways, and eventually by removing '@cucumber/ from import statements.

That then left us with 6 errors:

$ make
if [[ -d javascript ]]; then cd javascript && make default; fimake[1]: Entering directory '/home/<user>/src/cucumber/compatibility-kit/javascript'npm run build                                                          
                                                                            
> @cucumber/compatibility-kit@3.0.0 build
> tsc                                                                                                                            
features/attachments/attachments.ts:6:1 - error TS2554: Expected 2 arguments, but got 1.
                                                      
6 Before(() => undefined)      
  ~~~~~~~~~~~~~~~~~~~~~~~

  node_modules/fake-cucumber/dist/src/SupportCode.d.ts:6:53
    6 declare type RegisterHook = (tagExpression: string, body: AnyBody) => void;
                                                          ~~~~~~~~~~~~~
    An argument for 'body' was not provided.
                                                                                  
features/attachments/attachments.ts:6:8 - error TS7011: Function expression, which lacks retu
rn-type annotation, implicitly has an 'any' return type.
                                                      
6 Before(() => undefined)

features/data-tables/data-tables.ts:3:23 - error TS2307: Cannot find module 'fake-cucumber/di
st/src/DataTable' or its corresponding type declarations.

3 import DataTable from 'fake-cucumber/dist/src/DataTable'
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

features/hooks/hooks.ts:3:1 - error TS2554: Expected 2 arguments, but got 1.

3 Before(function () {
  ~~~~~~~~~~~~~~~~~~~~
4   // no-op
  ~~~~~~~~~~
5 })
  ~~

  node_modules/fake-cucumber/dist/src/SupportCode.d.ts:6:53
    6 declare type RegisterHook = (tagExpression: string, body: AnyBody) => void;
                                                          ~~~~~~~~~~~~~
    An argument for 'body' was not provided.

features/hooks/hooks.ts:15:1 - error TS2554: Expected 2 arguments, but got 1.

15 After(function () {
   ~~~~~~~~~~~~~~~~~~~
16   throw new Error('Exception in hook')
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 17 })
   ~~

  node_modules/fake-cucumber/dist/src/SupportCode.d.ts:6:53
    6 declare type RegisterHook = (tagExpression: string, body: AnyBody) => void;
                                                          ~~~~~~~~~~~~~
    An argument for 'body' was not provided.

features/parameter-types/parameter-types.ts:2:17 - error TS2305: Module '"fake-cucumber"' has
 no exported member 'ParameterType'.

2 import { Given, ParameterType } from 'fake-cucumber'
                  ~~~~~~~~~~~~~


Found 6 errors.

default.mk:25: recipe for target '.built' failed
make[1]: *** [.built] Error 2
make[1]: Leaving directory '/home/<user>/src/cucumber/compatibility-kit/javascript'
default.mk:17: recipe for target 'default-javascript' failed
make: *** [default-javascript] Error 2

which brings us here....

Steps to Reproduce (for bugs)

See above.

Context & Motivation

How has this issue affected you? Blocking progress on validating and document the Cucumber-Rust implementation level of compatibility.

What are you trying to accomplish? Get the Cucumber-Rust implementation to a point where it can be included upstream.

Your Environment

• Version used: git as at this date
• Operating System and version:
  • Ubuntu 18.04
  • node v15.11.0
  • npm 7.6.0
• Link to your project: https://github.com/bbqsrc/cucumber-rust

[luke-hill]

Just quickly (Might not matter), is there a reason you're not using fermium LTS 14? It would be a better node version to use for ensuring a wide variety of people use it / stability.

[bbros-dev]

is there a reason you're not using fermium LTS 14?

Sorry I know no node so don't even know what that means. We tried various node versions start with ubuntu's defualt, then went to v14 and v14 LTS from NodeSource, at version 15 we got actual progress, but still the errors shown. Very frustrating.

Can you point me to your docs where there are the required/minimum node setup instructions?

[davidjgoss]

@bbros-dev

Node issues

It can be painful to work across JavaScript packages in the monorepo at the moment. This is being hugely improved in #1372 which is close to being done I think.

The simplest workaround to get a particular package running as needed is to change the values for other @cucumber/ dependencies in the package.json from "file..." paths to just the latest published version and then doing npm install.

CCK

As far as I know, the monorepo doesn't contain any tools for running the CCK suite or producing reports. That's an exercise left to the cucumber implementation - run the features from the CCK with support code equivalent to the example JavaScript ones, and compare the messages that come out with what the CCK defines (with some normalisation to deal with ids, timestamps etc).

Examples:

• JavaScript: https://github.com/cucumber/cucumber-js/blob/master/compatibility/cck_spec.ts
• Java: https://github.com/cucumber/cucumber-jvm/blob/main/compatibility/src/test/java/io/cucumber/compatibility/CompatibilityTest.java

I think there are varying approaches between the implementations in terms of how the CCK features and messages are brought in:

• JavaScript: we publish @cucumber/compatibility-kit to npm and then import that in cucumber-js
• Java the features and messages are just copied to the cucumber-jvm repo
• Ruby, last I checked, was linking to the monorepo at a certain tag as a build step.

I'm not sure how the Rust ecosystem works in terms of dependency management, but maybe the fastest starting point would be the Java approach of just copy the files over and start building your test infra against them.

Paging @aslakhellesoy who may be able to add more and/or correct me

[bbros-dev]

Thanks @davidjgoss, appreciate the detailed response.

Paging @aslakhellesoy who may be able to add more and/or correct me

Unless there is something incorrect we have the info we need, and will wait for #1372.

Closing - "Update contributor docs" is on the checklist in #1372

[aslakhellesoy]

Hi @bbros-dev! I'm sorry this is so cryptic. @davidjgoss gave an excellent description - that's how it works. In my own words:

The Cucumber CCK is just a corpus of Feature files, step definitions + hooks and generated messages. The messages are generated by fake-cucumber, which acts as the reference implementation. (It's not as complete and nice to use as cucumber-js, but I don't think it would be very hard to make it a contender if we wanted to).

This corpus of files is used for Approval Testing of other Cucumber implementations.

Other Cucumber Implementation can use the CCK to verify its own message output with that of the CCK. This
CCK test suite also provides platform-specific implementations of the step definitions + hooks in the CCK.

It is not required that each implementation produce exactly the same output, but it should be equivalent.
We don't have a rigorous definition of what equivalent means, but we check the following (to the best of
my memory): The number of steps are the same, the number of step definitions are the same, the results are the same, etc.

And believe it or not, we also do some manual 😱 testing by feeding the generated messages through @cucumber/html-formatter, which is based on @cucumber/react. For this we use StoryBook. Using the StoryBook while you implement your Cucumber implementation's message output is actually a nice workflow with very useful and quick feedback.

[bbros-dev]

Thanks @aslakhellesoy, that also helped a lot in thinking about how best to use the CCK testdata.

Closing: Re-opened by mistake