Different test result with meson and cmake/ninja build?

[ClausKlein]

After cmake build, the default tests are OK

ninja 
....
Testing BuilderTest/settings: OK
Testing IteratorTest/convert: OK
Testing IteratorTest/decrement: OK
Testing IteratorTest/reverseIterator: OK
Testing IteratorTest/distance: OK
Testing IteratorTest/nullValues: OK
Testing IteratorTest/staticStringKey: OK
Testing IteratorTest/names: OK
Testing IteratorTest/indexes: OK
Testing IteratorTest/constness: OK
Testing RValueTest/moveConstruction: OK
Testing FuzzTest/fuzzDoesntCrash: OK
Testing MemberTemplateAs/BehavesSameAsNamedAs: OK
Testing MemberTemplateIs/BehavesSameAsNamedIs: OK
All 114 tests passed

To Reproduce

Claus-MBP:.build-jsoncpp-Debug clausklein$ ninja test
[0/1] Running tests...
Test project /Users/clausklein/Workspace/cpp/.build-jsoncpp-Debug
    Start 1: jsoncpp_readerwriter
1/3 Test #1: jsoncpp_readerwriter ................   Passed    4.36 sec
    Start 2: jsoncpp_readerwriter_json_checker
2/3 Test #2: jsoncpp_readerwriter_json_checker ...***Failed    1.73 sec
    Start 3: jsoncpp_test
3/3 Test #3: jsoncpp_test ........................   Passed    0.02 sec

67% tests passed, 1 tests failed out of 3

Total Test time (real) =   6.15 sec

The following tests FAILED:
	  2 - jsoncpp_readerwriter_json_checker (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util 
cd /Users/clausklein/Workspace/cpp/.build-jsoncpp-Debug && /opt/local/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
Claus-MBP:.build-jsoncpp-Debug clausklein$ 

Expected behavior

Claus-MBP:build clausklein$ ninja test
[0/1] Running all tests.
1/2 unittest_jsoncpp_test                   OK       0.03 s 
2/2 unittest_jsontestrunner                 OK       3.56 s 

Ok:                    2
Expected Fail:         0
Fail:                  0
Unexpected Pass:       0
Skipped:               0
Timeout:               0

Full log written to /Users/clausklein/Workspace/cpp/jsoncpp/build/meson-logs/testlog.txt
Claus-MBP:build clausklein$ 

Desktop (please complete the following information):

• OSX: Darwin Kernel Version 15.6.0
• commit edf528e (HEAD -> master, origin/master, origin/HEAD)
• ninja version ("1.9.0")

[dota17]

Yes, they are different. But the test result with meson/ninja build contains the one with cmake build, you can see this from meson.build.

[ClausKlein]

And what is right, should the ctest jsoncpp_readerwriter_json_checker fail?

If not, I hope, why is this test not included in the meson build test suite?

[dota17]

All test should not fail.
But from https://github.com/open-source-parsers/jsoncpp/blob/master/CONTRIBUTING.md#running-the-tests-manually, sometimes all test would not pass.

[cdunn2001]

There are 2 separate issues here:

1. JSON-checker tests will only pass in "strict-mode". They should not be run in non-strict mode, and that means they might not work on the old (pseudo deprecated) parser.
2. Cmake and Meson should run the same tests.

We'll be able to address both of those when we reorganize (and hopefully simplify) how the tests are run. Our first step in that direction is #1102.