Make check doesn't run tests in src/tests

[zbeekman]

AFAIK make check will build the package and also, I think, the tests, but doesn't actually execute them.

[zbeekman]

@blippy would you mind looking at this? AFAIK there was no make check before autotools came along (but I could be mistaken)

[blippy]

@zbeekman Sure, I'll take a look at it. I've never done a test harness before, but the way I understand it is that autotools uses a program's exit code to determine if there's been a pass or fail. So I may have to modify the test program to effect this.

[zbeekman]

I may be mistaken, but I didn't see anything running with make check. I saw it compile stuff, but the tests never ran. (Unless all their output was redirected.)

Don't worry about the stop code yet; most compilers respect Fortran stop # where # is some positive integer and use that as the exit code. I'm happy to look at what (if any) changes need to be applied to the test program. Just ensuring that it is actually compiled and run is enough of a start for me.

[blippy]

@zbeekman That's correct, nothing runs when you do make check. Autotools does have testing facilities, but I am not yet familiar with how to use them.

Stay tuned.

[blippy]

Added some test functionality as part of recent commits.

It's clunky in how it does it, but doing 'make check' should now work.

A better solution would be to have datetime_tests output lines:

test PASS empty datetime() constructor
test FAIL semi-empty datetime() constructor

instead of

test emtpy datetime() constructor       : PASS
test semi-empty datetime() constructor  : FAIL

and so on, and avoid the use of escape sequences. tests-env.sh massages the output, and it would be easier and more robust if it was in a more suitable form for processing.

[zbeekman]

@blippy 🙇 🎉

[blippy]

@zbeekman 👍

Ideally I think we should engineer the output format of datetime_tests so that not so much data munging is required. It works for now, I guess, which is the main thing.

I generally don't run 'make check' when I install software, although it light of some of the problems that are highlighted, it is a worthwhile exercise.

It took me awhile to figure out how to do the checking, as the autotools documentation was a bit on the sparse side. Apparently it can do parallel checking, but I'm not sure I want to go there.

[zbeekman]

@blippy fair. I just returned from the gym, and need to cleanup and run an errand, but then I plan on taking your PR for a spin. I think if that all looks good, I'll merge it (assuming @milancurcic doesn't have any objections, mint a new release, and then submit the Homebrew formula PR.

I agree with you about the tests, but maybe we can include a command line flag to the test program to include or exclude color escape sequences...

Also, what do you think about:

PASS: test emtpy datetime() constructor
FAIL: test semi-empty datetime() constructor

wouldn't that be even easier to parse?

[blippy]

Either way will work.

What happens is that datetime_tests get written to a log file. So anyone can have a look at the original output.

The testing script then takes the output and creates a trs file having lines like
:test-result: PASS foo ...

It looks as though autotools picks up these lines and performs a count.

[milancurcic]

Sounds great to me, thanks @blippy!!

@zbeekman yes, the plan is good, please go ahead with the merge and push when ready. Also please add yourself to CONTRIBUTORS.

[zbeekman]

The testing script then takes the output and creates a trs file having lines like
:test-result: PASS foo ...

It looks as though autotools picks up these lines and performs a count.

So, let's scrap the colon in the output I suggested, and then we can just prepend the :test-result: when we parse the test output. Nice work @blippy!