Acceptance tests failing on Travis

[glennblock]

After a recent checkin, the build failed on Travis running acceptance tests. Testing locally on Windows and OSX works fine. Seems to be related to Travis.

@adamralph any chance you can take a look at this?

[ilovezfs]

Same failure as is seen in the scriptcs Linux Travis CI is occurring on Jenkins for Homebrew in Homebrew/homebrew-core#2620.

ScriptCs.Tests.Acceptance.CommandLine.UnexpectedOption() [01.01.03] And I see an error message regarding the unknown option [FAIL]
   Should.Core.Exceptions.ContainsException : Assert.Contains() failure: Not found: unknownoption
   Stack Trace:
     at Xunit.Sdk.ExceptionUtility.RethrowWithNoStackTraceLoss (System.Exception ex) <0x1077ec910 + 0x000ed> in <filename unknown>:0 
     at Xbehave.Sdk.SyncStep.Execute () <0x10779bbe0 + 0x00288> in <filename unknown>:0 
     at Xbehave.Sdk.StepCommand.Execute (System.Object testClass) <0x10779b8c0 + 0x0013a> in <filename unknown>:0

etc. etc.

The only way I've found to avoid the failure is to use mono 4.2.x instead of 4.4.x, which isn't actually possible for Homebrew since we only have the latest version of mono in homebrew/core.

Maybe a newer version of xunit will avoid this?

[ilovezfs]

Here's a full build log https://gist.github.com/ilovezfs/c36095dd2054857dfdfcb9a22e1e3453

[glennblock]

Awesome. I have been meaning to look into why Travis is failing. I will
look into this.
On Mon, Sep 5, 2016 at 3:12 AM ilovezfs notifications@github.com wrote:

Here's a full build log
https://gist.github.com/ilovezfs/c36095dd2054857dfdfcb9a22e1e3453

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#1172 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAInRM4f3SmBxo3L5WgaeFGBrNM2Pwnrks5qm-rygaJpZM4JYgHa
.

[ilovezfs]

@glennblock thanks a lot! I was hoping to figure this out so I could merge my 0.16.1 PR, and then I realized the existing version in homebrew/core (0.16.0) is broken too, apparently ever since that mono upgrade. I'm not sure how that escaped notice at the time :(

[glennblock]

Working on this now, thanks @ilovezfs for the pointer.

[glennblock]

@ilovezfs I tried the latest xunit console runner (2.1.0) using the newer mono and it still fails.

In terms of brew, do you run the unit tests? scriptcs itself "should" work, the main issue appears to be something to do with with xunit or XBehave (which is the test framework we use which builds on xunit).

@bradwilson @adamralph any thoughts?

[bradwilson]

We do not officially support Mono for anything other than Android and iOS. The desktop CLR implementation in Mono has historically been filled with bugs triggered by xUnit.net (especially related to app domains and shadow copying). People have had some success when turning off shadow copying and/or app domains.

[glennblock]

@bradwilson ok, thanks for responding quickly and weighing in.

[ilovezfs]

For Homebrew, we do this:

  def install
    script_file = "scriptcs.sh"
    system "./build.sh"
    libexec.install Dir["src/ScriptCs/bin/Release/*"]
    (libexec/script_file).write <<-EOS.undent
      #!/bin/bash
      mono #{libexec}/scriptcs.exe $@
    EOS
    (libexec/script_file).chmod 0755
    bin.install_symlink libexec/script_file => "scriptcs"
  end

[glennblock]

OK so ./build.sh builds AND runs the unit tests. Is it necessary for tests to run? I can add a homebrew_build.sh for example that doesn't run the tests.

[ilovezfs]

No not necessary, but we typically don't merge anything whose test suite or make check/make test fails unless upstream itself says it's good-to-go despite that!

[glennblock]

I understand. I am trying to see if @adamralph can help me look into this one.

For now, I think I am gong to pin to an earlier mono just so I can get our travis tests to pass. I understand this doesn't unblock brew, but it will at least unblock our failing build.

Nothing has changed in our code, which means there is either a bug in mono, XUnit or XBehave.

[glennblock]

@ilovezfs going to try a few other things.

[ilovezfs]

Yeah, pinning to the earlier mono for your testing is a great idea until this is sorted out, but doesn't unblock Homebrew unless you guys say it's fine with the current mono despite that. A build script designated for Homebrew would unblock things temporarily, since there's no reason to believe we're making the situation worse given that what's already in core isn't working now, but I suspect you'll be able to figure out the problem soon regardless :)

[glennblock]

I am trying something that "may" solve it.

[ilovezfs]

@glennblock it's somewhat surprising it's still broken with the current xunit. Are you able to get a hello world for xunit working? Maybe there's something really simple broken before it even gets off the starting line.

[glennblock]

Most of the tests pass, only 18 fail, so xunit is definitely working.

[ilovezfs]

Are the ones failing all of the "expect failure" variety?

[glennblock]

Nope

[ilovezfs]

interesting, ok.

[glennblock]

This output looks pretty wacky:

ScriptCs.Tests.Acceptance.Configuration.CustomConfiguration() [01.01.03] When I execute the script without the log level option but specifying the custom config [FAIL]
  ScriptCs.Tests.Acceptance.Support.ScriptCsException : scriptcs.exe exited with code 2. The output was: 慃湮瑯漠数獡敳扭祬✠猯捲匯牣灩䍴⽳楢⽮敒敬獡⽥捳楲瑰獣攮數㨧丠畳档映汩⁥牯搠物捥潴祲ਮ

[ilovezfs]

unicode settings broken maybe?

[glennblock]

Not sure. The way these acceptance tests work, is each test launches scriptcs (fires off the process programmaticly) and then parses the output returned and matches against expected output.

[glennblock]

My guess is the issue is in XBehave not in xunit, which the stack trace seems to indicate. I am not positive though.

[adamralph]

As you say @glennblock, I think all the tests which are failing are those which launch scriptcs.exe, capture the console output and then assert on the content.

I doubt this is an xbehave problem. It's probably at a lower level than that. The way to prove that would be to temporarily lower one of the acceptance test methods from an xbehave scenario down to an xunit fact, and see if it still fails.

[glennblock]

@adamralph good idea. I'll try that.

[glennblock]

@adamralph don't all our acceptance tests do that though? Out of like 80, only 18 are failing.

[glennblock]

Hmm, there's one other possibility. The "should" library.

[adamralph]

With xbehave, each "test" is a step within a scenario, so "82 total, 17 failed" means that were 82 steps in total, of which only 17 failed. E.g. in

"Given a hello world script"
    .f(() => directory = ScenarioDirectory.Create(scenario)
        .WriteLine("foo.csx", @"Console.WriteLine(""Hello world!"");"));

"When I execute the script with debug set to {0}"
    .f(() => output = ScriptCsExe.Run("foo.csx", debug, directory));

"Then I see 'Hello world!'"
    .f(() => output.ShouldContain("Hello world!"));

only the last step will be failing, the first two should be passing.

[glennblock]

Aaah. Which could support my theory of Should.

[glennblock]

I am going to try dropping .ShouldContain on the first scenario and replace it with an Assert

[adamralph]

Should is only asserting that a string contains some other string so I'd be surprised if the problem was there (but then Mono is full of surprises). My suspicion is that something has changed in Mono regarding stdout capture from a process.

[glennblock]

That does sound more reasonable.

[glennblock]

I am going to add a console.writeline to the output.

[glennblock]

Thank you @adamralph!

[glennblock]

@adamralph or the other possibility is it is not passing IN the arguments properly.

[glennblock]

I think you may be right, there may be something broken with reading the output stream, looking at what we are doing here: https://github.com/scriptcs/scriptcs/blob/dev/test/ScriptCs.Tests.Acceptance/Support/ProcessStartInfoExtensions.cs

[adamralph]

Yep, some weird change in System.Diagnostics in the latest Mono is my chief suspect.

[glennblock]

Yup, now that I've had a few hours on this and the fact that it works with earlier Mono builds is pointing at this being a Mono bug. I'll file something in the Mono bug tracker.