Don't report expected failures as failures to Travis; added #version:…

[theseion]

… property to the Metacello load spec

• the test report would report run as failure even if only expected failures had actually failed
• added #version: accessor to Metacello load spec. This allows one to specify a specific version that should be loaded when using a ConfigurationOf invocation.

[fniephaus]

Thanks! I may have time to go over this tomorrow evening.
Is this Squeak only?

[theseion]

The expected failures fix is for Squeak only. The addition of #version: concerns both Squeak and Pharo.

[fniephaus]

That makes sense.
What's the reason for the self halt.s? https://github.com/theseion/smalltalkCI/blob/9564df37c9d2c99abc94844136b1b57e6a8ff96a/src/SmalltalkCI-Squeak.st#L259-L267

[fniephaus]

Re expected failures: since this can be quite confusing, I had already added these tests:
https://github.com/hpi-swa/smalltalkCI/tree/b63fb728df06b0b4f7e224554f4b917be0b62d21/tests/SmalltalkCI-Excluded-Tests.package

There's a test called testShouldPassUnexpectedly which is expected to fail, but it passes. So overall, this test should fail, right? With the current version, it actually fails. With your version, it does not. In fact, I think your version is the one I found in a recent Pharo image, so Pharo behaves exactly like your version. I would assume that my version is wrong then, but maybe this is a Pharo bug? If a test is expected to fail, it should not pass, right?

Since both of your changes are in one commit, I will need to wait until we have discussed this before I can merge this.

[fniephaus]

This PR is also related to #47

[fniephaus]

Re #60 (comment): now I get the problem! TestFailures were always written to the log, even if it they were expected. Also, your version is nicer, because it's less nested. Merging...

Lastly, Pharo's code is different and I'm afraid we found a bug...

[theseion]

What's the reason for the self halt.s?

Wow! Sorry about those. Simply forgot them.

In Pharo the philosophy of an expected failure seems to be that if a test fails then that is the expected outcome. In other words if the test doesn't fail then something has changed and the outcome is unexpected. A failure will then alert to a changed conditions.
I'm not saying that this is the best possible way to understand expected failures but it is a definition I have come to find convenient.
In Fuel we have a couple of test cases that we know are going to fail because they cover things that can't work (e.g. because of architectural constraints) and we use them for documentation. I don't want those tests to show up as failures because to a user they convey the impression that something is broken when really everything's fine.

So now I have all my beautifully green Fuel builds but Travis marks the (Squeak) builds as failures because the reporter exits with an error code.

If you look at the Squeak TestRunner, it will produce exactly that output by the way, exactly the same as in Pharo.

[theseion]

What's the bug in Pharo?

[fniephaus]

Any way we can have a quick chat about this in Pharo's Slack or IRC?

[fniephaus]

A test passing, even though it's marked as an expected failure, does not output anything in Pharo's HDTestReport. That's the bug I was talking about 😃

[dalehenrich]

I saw the same problems in the test runner when I ported it to GemStone. I fixed that behavior along with a couple of other things ... you know because these files are .st files and not a filetree repository we will never be able to share common code and we will continue to needlessly duplicate effort ...

[fniephaus]

@dalehenrich yes, that is a problem. Unfortunately, Filetree does not ship with Squeak images. It'd be much easier if it did! I have just "gittified" the .st files (c14cbb2), so that I can see a normal diff and use GitHub's web editor. This makes things better, but not perfect.
If I think about it, I could just load Filetree in the build process for Squeak test images and then load smalltalkCI via Filetree in Squeak, Pharo and GemStone. That would make things even easier!