Failed tests when checking implementation against common cucumber test suite

[erichstuder]

👓 What did I see?

I try to build cucumber-cpp according to the README.md
I have installed all the dependencies correctly i think. see my Docker-Container
The Container has not yet any dependencies to my project, so it can be run on its own.

FROM debian:12.1

WORKDIR /usr/src/cucumber-cpp

RUN apt-get update && \
    apt-get -y install cmake build-essential \
                       libboost-thread-dev libboost-system-dev \
                       libboost-regex-dev libboost-date-time-dev \
                       libboost-program-options-dev \
                       libboost-test-dev \
                       git \
                       ruby ruby-dev \
                       libboost-filesystem-dev 

RUN git clone https://github.com/cucumber/cucumber-cpp.git . && \
    git checkout 2f7d9ad519290a8b4a6ee6845af4480dd9ec2b7f

RUN gem install bundler && \
    bundle install

Then in the container I build cucumber-cpp as advised:

# Download test suite
git submodule init
git submodule update

# Create build directory
cmake -E make_directory build

# Generate Makefiles
cmake -E chdir build cmake -DCUKE_ENABLE_EXAMPLES=on -DCMAKE_INSTALL_PREFIX=${prefix} ..

# Build cucumber-cpp and tests
cmake --build build

# Run unit tests
cmake --build build --target test

# Run install
cmake --build build --target install

# Check implementation against common cucumber test suite
cmake --build build --target features

In the last step cmake --build build --target features I see a lot of failed tests:

*** THIS RUBY IMPLEMENTATION DOESN'T REPORT FILE AND LINE FOR PROCS ***
F---F-----F------F------F-----F---F---F---F---F---F---F---F---F-----F---F---F---F-----F--------F-----F---F-----F---F---F---F---F---F---F----F----F-----F----F------F------F-----F-----F----F--------F----------F----------F---F---F----F-----F----F-------------------------F----------------------------------------FF----F----

Failing Scenarios:
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/comments.feature:5 # Scenario: Comments in a feature file
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:16 # Scenario: All steps passing means the scenario passes
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:27 # Scenario: Failing step means the scenario fails
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:39 # Scenario: Pending step means the scenario is pending
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:51 # Scenario: Missing step mapping means the scenario is undefined
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:62 # Scenario: Feature headers
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:73 # Scenario: Simple flat steps
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:83 # Scenario: Given, When, Then, And and But steps
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:95 # Scenario: Failing steps
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:105 # Scenario: Single-parameter step
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:115 # Scenario: Two-parameter step
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:125 # Scenario: Two-parameter step failing
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/core.feature:135 # Scenario: Three-parameter step
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/data_tables.feature:3 # Scenario: a data table interpreted as an array
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/data_tables.feature:21 # Scenario: a data table can be read as an array of hashes
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/data_tables.feature:37 # Scenario: a data table can be read as an array of values
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/doc_strings.feature:8 # Scenario: Step accepting a DocString parameter
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/failing_steps.feature:6 # Scenario: see exception message
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/failing_steps.feature:17 # Scenario: see failing scenarios
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:8 # Scenario: Before and After
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:15 # Scenario: Around
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:20 # Scenario: Around fires around other hooks
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:27 # Scenario: Tagged hook with matching scenario tag
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:32 # Scenario: Tagged hook without matching scenario tag
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:37 # Scenario: Tagged hook with untagged scenario
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/hooks.feature:42 # Scenario: Untagged hook with tagged scenario
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/i18n.feature:9 # Scenario: A non-english Feature
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/i18n.feature:23 # Scenario: Only One Language per File
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:3 # Scenario: execute scenarios matching a tag
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:10 # Scenario: execute scenarios not matching a tag
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:17 # Scenario: execute scenarios matching any of several tags (OR)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:25 # Scenario: execute scenarios matching several tags (AND)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:32 # Scenario: execute scenarios not matching any tag (NOT OR NOT)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:41 # Scenario: exclude scenarios matching two tags (NOT AND NOT)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:50 # Scenario: with tag or without other tag
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:58 # Scenario: with tag but without two other tags
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/tags.feature:66 # Scenario: execute scenario with tagged feature
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/undefined_steps.feature:6 # Scenario: step definition snippet suggestion
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/undefined_steps.feature:21 # Scenario: repeating steps (And, But)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/undefined_steps.feature:42 # Scenario: star steps (*)
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/undefined_steps.feature:68 # Scenario: step with doc strings
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/undefined_steps.feature:81 # Scenario: step with data tables:
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/world.feature:3 # Scenario: access World variable
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/world.feature:9 # Scenario: different World instances between scenarios
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/world.feature:16 # Scenario: call function from World
cucumber /usr/src/cucumber-cpp/features/cucumber-tck/world.feature:22 # Scenario: replace World constructor
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:27 # Scenario Outline: <check> macro, Examples (#1)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:28 # Scenario Outline: <check> macro, Examples (#2)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:29 # Scenario Outline: <check> macro, Examples (#3)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:30 # Scenario Outline: <check> macro, Examples (#4)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:31 # Scenario Outline: <check> macro, Examples (#5)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:32 # Scenario Outline: <check> macro, Examples (#6)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:33 # Scenario Outline: <check> macro, Examples (#7)
cucumber /usr/src/cucumber-cpp/features/specific/boost_specific.feature:34 # Scenario Outline: <check> macro, Examples (#8)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:29 # Scenario Outline: <characters> in step definition, Examples (#1)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:30 # Scenario Outline: <characters> in step definition, Examples (#2)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:31 # Scenario Outline: <characters> in step definition, Examples (#3)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:32 # Scenario Outline: <characters> in step definition, Examples (#4)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:33 # Scenario Outline: <characters> in step definition, Examples (#5)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:34 # Scenario Outline: <characters> in step definition, Examples (#6)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:35 # Scenario Outline: <characters> in step definition, Examples (#7)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:36 # Scenario Outline: <characters> in step definition, Examples (#8)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:37 # Scenario Outline: <characters> in step definition, Examples (#9)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:38 # Scenario Outline: <characters> in step definition, Examples (#10)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:39 # Scenario Outline: <characters> in step definition, Examples (#11)
cucumber /usr/src/cucumber-cpp/features/specific/escaping.feature:40 # Scenario Outline: <characters> in step definition, Examples (#12)
cucumber /usr/src/cucumber-cpp/features/specific/simple.feature:7 # Scenario: Simple Scenario
cucumber /usr/src/cucumber-cpp/features/specific/wire_encoding.feature:6 # Scenario: Multibyte Character Step Matching

68 scenarios (68 failed)
290 steps (290 skipped)
0m0.379s
gmake[3]: *** [CMakeFiles/features.dir/build.make:70: CMakeFiles/features] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:304: CMakeFiles/features.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:311: CMakeFiles/features.dir/rule] Error 2
gmake: *** [Makefile:205: features] Error 2

The Calc example runs without errors:

build/examples/Calc/BoostCalculatorSteps >/dev/null &
cucumber examples/Calc

✅ What did I expect to see?

Actually I don't know what exactly to expect.
I find it akward that there are failed tests.
So probably I should expect no failed tests?

📦 Which tool/library version am I using?

Docker Container with Debian 12.1 as you can see above
I tried it on a Linux and Windows with WSL with the same outcome.

🔬 How could you reproduce it?

1. run the Docker file from bash with script below or just run run_cucumber.sh
   this will setup and run the container

TAG=$(echo ${PWD:1} | tr '[:upper:]' '[:lower:]')
WORKDIR=/usr/src/cucumber-cpp
docker build -t $TAG .
docker run \
	--rm \
	--name cucumber \
	--workdir ${WORKDIR} \
	-it $TAG \
	bash

2. execute the steps to build and install cucumber-cpp

# Download test suite
git submodule init
git submodule update

# Create build directory
cmake -E make_directory build

# Generate Makefiles
cmake -E chdir build cmake -DCUKE_ENABLE_EXAMPLES=on -DCMAKE_INSTALL_PREFIX=${prefix} ..

# Build cucumber-cpp and tests
cmake --build build

# Run unit tests
cmake --build build --target test

# Run install
cmake --build build --target install

# Check implementation against common cucumber test suite
cmake --build build --target features

📚 Any additional context?

Let me know if you need more information.

[ursfassler]

Hi @erichstuder
unfortunately the e2e test don't work anymore. They use the Cucumber Technology Compatibility Kit which will no longer be developed and depends on quite old packages.
We plan to replace or remove the e2e test.

[ursfassler]

I removed the broken e2e tests with #272.