Travis - OSX CI

[konserw]

My try on enabling OSX CI on travis. There are 2 issues:

1. gtest 1.6.0 does not build - tried a few things i could find on internet but failed
2. there are a few (3?) test from target features that fail, so shall we:
   a) remove that target from osx ci
   b) fix it somehow
   c) drop cucumber-tck altogether?

[paoloambrosio]

1. Not a problem: GTest 1.6 is tested with Linux
2. Did you understand why that test is failing? Are you sure it's not failing because of a bug?
   a) only if option b is not possible after finding why it's failing
   b) yep, that's usually what should be done when tests fail!
   c) please note that this means rewriting the entire functional test suite before removing Cucumber-TCK... that is a massive effort; I'd rather wait till the Cucumber Team collectively comes up with a better solution (might have a direction in April)

[konserw]

1. ok then
2. This is black magic to me honestly :) Have you checked logs? It looks like those steps were undefined, what is impossible as the same test passes on linux...

[konserw]

Ok - I honestly dont know what is going on - I've pushed one commit just to enable building on my travis and in result: my travis reported 6 errors and cucumber travis (with the same code) reported 3 errors, when there was only one error with previous build (with the same code...)

[konserw]

Now after rerun - 5 fails

[MartinDelille]

@konserw when running cmake --build build --target features locally I also have 5 fails (under OSX 10.9)

[MartinDelille]

In fact the number varies upon launch, now I have 8 fails.

[paoloambrosio]

Glad to know it fails locally on OSX, so it can be "easily" debugged. The randomness of failures make me think that Cucumber-CPP doesn't have time to create the socket before Cucumber-Ruby tries to connect.

In features/step_definitions/cucumber_cpp_mappings.rb we run Cucumber-CPP without waiting for the port to be open:

  def run_cucumber_cpp
    @steps_out = IO.popen STEP_DEFINITIONS_EXE
  end

To verify if this is the problem, a "sleep 1" before the end would do, but that's not a solution.

Ideally I'd wait for a message to be displayed but there are no log messages. We could introduce in src/main.cpp another command line parameter "-v" to trigger verbose mode and write a log line after listening on the port like...

    server.listen(port);
    if (verbose)
        std::cerr << "Listening on port " << port << std::endl;
    server.acceptOnce();

...then changing the Ruby method run_cucumber_cpp to wait for that log line to be displayed. As in all tests, if that does not happen in a reasonable amount of time, it should fail. It would take a while with my Ruby skills, hope someone elses are better ;-)

[MartinDelille]

I just tried adding the sleep 1 line to the run_cucumber_cpp method and the fails step disappeared.

[paoloambrosio]

This SHOULD (not tested!) wait at most one second for output on either standard output or error, and verifies that it starts with Listening:

require "io/wait" # it might be already included

...

  def run_cucumber_cpp
    @steps_out = IO.popen STEP_DEFINITIONS_EXE
    expect(@steps_out.wait(1)).not_to be_nil
    expect(@steps_out.readline).to start_with("Listening")
  end

[paoloambrosio]

The bind: Address already in use message is because my solution reads only from std::cout and not from std::cerr :-/ The standard error is just printed. Guess the only option is Open3.popen3 (http://mentalized.net/journal/2010/03/08/5-ways-to-run-commands-from-ruby/). I've applied a change in local to master and it works. I'll try and create a branch ASAP with the required changes for @konserw to include it in the PR.

[paoloambrosio]

Here it is: paoloambrosio/cucumber-cpp@757796a.

[konserw]

I've just changed std::cerr in main.cpp to std::cout yesterday,
but OK, I'll check your solution.

[paoloambrosio]

I should have used clog to be correct, even if both clog and cerr are redirected to the stderr.

[MartinDelille]

@paoloambrosio I checked out paoloambrosio/cucumber-cpp@757796a but I have the following error:

When Cucumber runs the feature     # /Users/martin/dev/demo/cucumber-cpp/features/cucumber-tck/step_definitions/cucumber_stepdefs.rb:93
  expected "bind: Address already in use\n" to start with "Listening" (RSpec::Expectations::ExpectationNotMetError)
  /Users/martin/dev/demo/cucumber-cpp/features/step_definitions/cucumber_cpp_mappings.rb:326:in `run_cucumber_cpp'
  /Users/martin/dev/demo/cucumber-cpp/features/step_definitions/cucumber_cpp_mappings.rb:295:in `run_feature_with_params'
  /Users/martin/dev/demo/cucumber-cpp/features/step_definitions/cucumber_cpp_mappings.rb:210:in `run_feature'
  /Users/martin/dev/demo/cucumber-cpp/features/cucumber-tck/step_definitions/cucumber_stepdefs.rb:94:in `/^Cucumber (?:runs|executes) the (?:feature|scenario)$/'
  /Users/martin/dev/demo/cucumber-cpp/features/cucumber-tck/comments.feature:20:in `When Cucumber runs the feature'

[konserw]

So maybe my solution with just sleep on osx is best?

[paoloambrosio]

Don't pull my changes in yours yet but sleep is not the right option when testing async systems. I'll try and run them on OSX to find what is wrong.

[konserw]

above you can see tries using PTY to execute cucumber-cpp - all failed

[MartinDelille]

I just realized that if your solution (paoloambrosio/cucumber-cpp@757796a) failed was because I still had a processus running. When re-running the feature target, I had the same result that before: a variable number of test failing.

[konserw]

I'm closing this PR as I've created better: #104