self-test problems

[0-wiz-0]

I've built meson-0.33.0 on NetBSD-7.99.35/amd64 with python-3.5.2.
When running run_tests.py, I see:

cd meson-0.33.0 && /usr/pkg/bin/python3.5 run_tests.py

Running common tests.

Succeeded test: test cases/common/1 trivial
Succeeded test: test cases/common/2 cpp
Succeeded test: test cases/common/3 static
Succeeded test: test cases/common/4 shared
Succeeded test: test cases/common/5 linkstatic
Failed test: test cases/common/6 linkshared
Reason: Running unit tests failed.
Succeeded test: test cases/common/7 mixed
Succeeded test: test cases/common/8 install
Succeeded test: test cases/common/9 header install
Succeeded test: test cases/common/10 man install
Succeeded test: test cases/common/11 subdir
Succeeded test: test cases/common/12 data
Failed test: test cases/common/13 pch
Reason: Compiling source code failed.
Failed test: test cases/common/14 cpp pch
Reason: Compiling source code failed.
Failed test: test cases/common/15 mixed pch
Reason: Compiling source code failed.
Failed test: test cases/common/16 configure file
Reason: Generating the build system failed.
Succeeded test: test cases/common/17 if
Succeeded test: test cases/common/18 else
Succeeded test: test cases/common/19 comparison
Succeeded test: test cases/common/20 array
Succeeded test: test cases/common/21 includedir
Succeeded test: test cases/common/22 header in file list
Succeeded test: test cases/common/23 global arg
Succeeded test: test cases/common/24 target arg
Succeeded test: test cases/common/25 object extraction
Succeeded test: test cases/common/26 endian
Succeeded test: test cases/common/27 library versions
Succeeded test: test cases/common/28 config subdir
Succeeded test: test cases/common/29 pipeline
Succeeded test: test cases/common/30 pipeline
Succeeded test: test cases/common/31 find program
Succeeded test: test cases/common/32 multiline string
Succeeded test: test cases/common/33 try compile
Succeeded test: test cases/common/34 compiler id
Succeeded test: test cases/common/35 sizeof
Succeeded test: test cases/common/36 define10
Succeeded test: test cases/common/37 has header
Succeeded test: test cases/common/38 run program
Succeeded test: test cases/common/39 tryrun
Succeeded test: test cases/common/40 logic ops
Succeeded test: test cases/common/41 elif
Succeeded test: test cases/common/42 string operations
Succeeded test: test cases/common/43 has function
Succeeded test: test cases/common/44 has member
Succeeded test: test cases/common/45 alignment
Failed test: test cases/common/46 library chain
Reason: Running unit tests failed.
Succeeded test: test cases/common/47 options
Failed test: test cases/common/48 test args
Reason: Running unit tests failed.
Failed test: test cases/common/49 subproject
Reason: Running unit tests failed.
Succeeded test: test cases/common/50 subproject options
Succeeded test: test cases/common/51 pkgconfig-gen
Succeeded test: test cases/common/52 custom install dirs
Failed test: test cases/common/53 subproject subproject
Reason: Running unit tests failed.
Succeeded test: test cases/common/54 same file name
Succeeded test: test cases/common/55 file grabber
Failed test: test cases/common/56 custom target
Reason: Generating the build system failed.
Failed test: test cases/common/57 custom target chain
Reason: Generating the build system failed.
Failed test: test cases/common/58 run target
Reason: Generating the build system failed.
Failed test: test cases/common/59 object generator
Reason: Generating the build system failed.
Succeeded test: test cases/common/60 install script
Failed test: test cases/common/61 custom target source output
Reason: Compiling source code failed.
Failed test: test cases/common/62 exe static shared
Reason: Running unit tests failed.
Succeeded test: test cases/common/63 array methods
Failed test: test cases/common/64 custom header generator
Reason: Compiling source code failed.
Failed test: test cases/common/65 multiple generators
Reason: Compiling source code failed.
Succeeded test: test cases/common/66 install subdir
Succeeded test: test cases/common/67 foreach
Succeeded test: test cases/common/68 number arithmetic
Succeeded test: test cases/common/69 string arithmetic
Succeeded test: test cases/common/70 array arithmetic
Succeeded test: test cases/common/71 arithmetic bidmas
Failed test: test cases/common/72 build always
Reason: Compiling source code failed.
Succeeded test: test cases/common/73 vcstag
Succeeded test: test cases/common/74 modules
Succeeded test: test cases/common/75 should fail
Failed test: test cases/common/76 configure file in custom target
Reason: Generating the build system failed.
Failed test: test cases/common/77 external test program
Reason: Running unit tests failed.
Failed test: test cases/common/78 ctarget dependency
Reason: Compiling source code failed.
Failed test: test cases/common/79 shared subproject
Reason: Running unit tests failed.
Failed test: test cases/common/80 shared subproject 2
Reason: Running unit tests failed.
Succeeded test: test cases/common/81 file object
Failed test: test cases/common/82 custom subproject dir
Reason: Running unit tests failed.
Succeeded test: test cases/common/83 has type
Succeeded test: test cases/common/84 extract from nested subdir
Succeeded test: test cases/common/85 internal dependency
Failed test: test cases/common/86 same basename
Reason: Running unit tests failed.
Succeeded test: test cases/common/87 declare dep
Succeeded test: test cases/common/88 extract all
Succeeded test: test cases/common/89 add language
Succeeded test: test cases/common/90 identical target name in subproject
Succeeded test: test cases/common/91 plusassign
Succeeded test: test cases/common/92 skip subdir
Failed test: test cases/common/93 private include
Reason: Compiling source code failed.
Succeeded test: test cases/common/94 default options
env: python3: No such file or directory
env: python3: No such file or directory
Succeeded test: test cases/common/95 dep fallback
Failed test: test cases/common/96 default library
Reason: Running unit tests failed.
Succeeded test: test cases/common/97 selfbuilt custom
Failed test: test cases/common/98 gen extra
Reason: Compiling source code failed.
Succeeded test: test cases/common/99 benchmark
Succeeded test: test cases/common/100 test workdir
Succeeded test: test cases/common/101 suites
Succeeded test: test cases/common/102 threads
Failed test: test cases/common/103 manygen
Reason: Compiling source code failed.
Succeeded test: test cases/common/104 stringdef
Failed test: test cases/common/105 find program path
Reason: Generating the build system failed.
Succeeded test: test cases/common/106 subproject subdir
Failed test: test cases/common/107 postconf
Reason: Generating the build system failed.
Failed test: test cases/common/108 postconf with args
Reason: Generating the build system failed.
Succeeded test: test cases/common/109 testframework options
Succeeded test: test cases/common/110 extract same name
Succeeded test: test cases/common/111 has header symbol
Succeeded test: test cases/common/112 has arg
Failed test: test cases/common/113 generatorcustom
Reason: Compiling source code failed.
Succeeded test: test cases/common/114 multiple dir configure file
Succeeded test: test cases/common/115 spaces backslash

Running failing tests.

Succeeded test: test cases/failing/1 project not first
Succeeded test: test cases/failing/2 missing file
Succeeded test: test cases/failing/3 missing subdir
Succeeded test: test cases/failing/4 missing meson.build
Succeeded test: test cases/failing/5 misplaced option
Succeeded test: test cases/failing/6 missing incdir
Succeeded test: test cases/failing/7 go to subproject
Succeeded test: test cases/failing/8 recursive
Succeeded test: test cases/failing/9 missing extra file
Succeeded test: test cases/failing/10 out of bounds
Succeeded test: test cases/failing/11 object arithmetic
Succeeded test: test cases/failing/12 string arithmetic
Succeeded test: test cases/failing/13 array arithmetic
Succeeded test: test cases/failing/14 invalid option name
Succeeded test: test cases/failing/15 kwarg before arg
Succeeded test: test cases/failing/16 extract from subproject
Succeeded test: test cases/failing/17 same name
Succeeded test: test cases/failing/18 wrong plusassign
Succeeded test: test cases/failing/19 target clash
Succeeded test: test cases/failing/20 version
Succeeded test: test cases/failing/21 subver
Succeeded test: test cases/failing/22 assert
Succeeded test: test cases/failing/23 rel testdir
Succeeded test: test cases/failing/25 int conversion
Succeeded test: test cases/failing/26 badlang
Succeeded test: test cases/failing/27 output subdir
Succeeded test: test cases/failing/28 noprog use
Succeeded test: test cases/failing/29 no crossprop

Running prebuilt tests.

Succeeded test: test cases/prebuilt/1 object
Succeeded test: test cases/prebuilt/2 static

Not running platform-osx tests.

Skipping: test cases/osx/1 basic
Skipping: test cases/osx/2 library versions

Not running platform-windows tests.

Skipping: test cases/windows/1 basic
Skipping: test cases/windows/2 winmain
Skipping: test cases/windows/3 cpp
Skipping: test cases/windows/4 winmaincpp
Skipping: test cases/windows/5 resources
Skipping: test cases/windows/6 vs module defs
Skipping: test cases/windows/7 mingw dll versioning
Skipping: test cases/windows/8 msvc dll versioning
Skipping: test cases/windows/9 find program

Running platform-linux tests.

Failed test: test cases/linuxlike/1 pkg-config
Reason: Generating the build system failed.
Succeeded test: test cases/linuxlike/2 external library
Failed test: test cases/linuxlike/3 linker script
Reason: Running unit tests failed.
Failed test: test cases/linuxlike/4 extdep static lib
Reason: Generating the build system failed.
Failed test: test cases/linuxlike/5 dependency versions
Reason: Generating the build system failed.
Failed test: test cases/linuxlike/6 subdir include order
Reason: Generating the build system failed.
Succeeded test: test cases/linuxlike/7 library versions
Succeeded test: test cases/linuxlike/8 subproject library install

Running framework tests.

Failed test: test cases/frameworks/1 boost
Reason: Generating the build system failed.
Failed test: test cases/frameworks/2 gtest
Reason: Generating the build system failed.
Failed test: test cases/frameworks/3 gmock
Reason: Generating the build system failed.
Failed test: test cases/frameworks/4 qt5
Reason: Generating the build system failed.
Failed test: test cases/frameworks/5 protocol buffers
Reason: Generating the build system failed.
Failed test: test cases/frameworks/6 gettext
Reason: Compiling source code failed.
Failed test: test cases/frameworks/7 gnome
Reason: Generating the build system failed.
Failed test: test cases/frameworks/8 flex
Reason: Generating the build system failed.
Failed test: test cases/frameworks/9 wxwidgets
Reason: Generating the build system failed.
Succeeded test: test cases/frameworks/10 gtk-doc

Not running java tests.

Skipping: test cases/java/1 basic
Skipping: test cases/java/2 subdir

Not running C# tests.

Skipping: test cases/csharp/1 basic
Skipping: test cases/csharp/2 library
Skipping: test cases/csharp/3 resource

Not running vala tests.

Skipping: test cases/vala/1 basic
Skipping: test cases/vala/2 multiple files
Skipping: test cases/vala/3 dep
Skipping: test cases/vala/4 config
Skipping: test cases/vala/5 target glib
Skipping: test cases/vala/6 static library
Skipping: test cases/vala/7 shared library

Not running rust tests.

Skipping: test cases/rust/1 basic
Skipping: test cases/rust/2 sharedlib
Skipping: test cases/rust/3 staticlib

Running objective c tests.

g++: error trying to exec 'cc1objplus': execvp: No such file or directory
Succeeded test: test cases/objc/1 simple
Failed test: test cases/objc/2 nsstring
Reason: Generating the build system failed.
Failed test: test cases/objc/3 objc++
Reason: Generating the build system failed.

Running fortran tests.

Failed test: test cases/fortran/1 basic
Reason: Generating the build system failed.
Failed test: test cases/fortran/2 modules
Reason: Generating the build system failed.
Failed test: test cases/fortran/3 module procedure
Reason: Generating the build system failed.
Failed test: test cases/fortran/4 self dependency
Reason: Generating the build system failed.
Failed test: test cases/fortran/5 static
Reason: Generating the build system failed.
Failed test: test cases/fortran/6 dynamic
Reason: Generating the build system failed.

Not running swift tests.

Skipping: test cases/swift/1 exe
Skipping: test cases/swift/2 multifile
Skipping: test cases/swift/3 library
Skipping: test cases/swift/4 generate
Skipping: test cases/swift/5 mixed

Not running python3 tests.

Skipping: test cases/python3/1 basic
Skipping: test cases/python3/2 extmodule
Skipping: test cases/python3/3 cython

Total configuration time: 30.32s
Total build time: 23.61s
Total test time: 2.22s

Total passed tests: 117
Total failed tests: 55
Total skipped tests: 34
*** Error code 55

Please advice how to get more details.
I guess at least some of these are just missing dependencies.

[jpakkane]

Meson captures all output during test run and writes it to files called meson-test-run.{txt, xml}. You should be able to get reasons for the tests failing from there.

You can also run any test yourself with these commands:

mkdir build
./meson.py <path to test case in question> build
ninja -C build

[0-wiz-0]

Ok, here are some details:

[1/3] Precompiling header '../test cases/common/13 pch/pch/prog.h'
[2/3] Compiling c object 'prog@exe/prog.c.o'
FAILED: prog@exe/prog.c.o 
cc  '-Iprog@exe' '-fdiagnostics-color=always' '-I../test cases/common/13 pch' '-I.' '-pipe' '-Wall' '-Winvalid-pch' '-O0' '-g' '-Iprog@exe' '-include' 'prog.h' '-MMD' '-MQ' 'prog@ex
e/prog.c.o' '-MF' 'prog@exe/prog.c.o.d' -o 'prog@exe/prog.c.o' -c '../test cases/common/13 pch/prog.c'
cc1: warning: prog@exe/prog.h.gch: had text segment at different address
cc1: error: one or more PCH files were found, but they were invalid
cc1: fatal error: prog.h: No such file or directory
compilation terminated.
ninja: build stopped: subcommand failed.

(similar for other pch tests)

test cases/common/16 configure file
....
Program "python3" not found.

I only have python3.5 (same problem for 61, 64, 65, 72, 78, 93, 98, 103, 105, 113, ; similarly 76 looks for python).

test cases/frameworks/6 gettext
...
[2/2] Linking target src/intlprog
FAILED: src/intlprog 
cc    -o src/intlprog 'src/intlprog@exe/intlmain.c.o' '-Wl,--no-undefined' 
src/intlprog@exe/intlmain.c.o: In function `main':
/scratch/wip/py-meson/work/meson-0.33.0/b viph4lby/../test cases/frameworks/6 gettext/src/intlmain.c:13: undefined reference to `bindtextdomain'
/scratch/wip/py-meson/work/meson-0.33.0/b viph4lby/../test cases/frameworks/6 gettext/src/intlmain.c:14: undefined reference to `textdomain'
/scratch/wip/py-meson/work/meson-0.33.0/b viph4lby/../test cases/frameworks/6 gettext/src/intlmain.c:15: undefined reference to `gettext'
ninja: build stopped: subcommand failed.

These symbols are in libintl (also on OS X and some other systems).

frameworks/7 gnome expects pkg-config to be installed.

framework/8 flex expects bison.

frameworks/9 wxwidgets expects wx-config-3.0 or wx-config.

The fortran tests fail with

Meson encountered an error in file meson.build, line 1, column 0:ESC[0m
Unknown compiler(s): "gfortran, g95, f95, f90, f77"
The follow exceptions were encountered:
Running "f95 --version" gave "[Errno 2] No such file or directory: 'f95'"
Running "f90 -V" gave "[Errno 2] No such file or directory: 'f90'"
Running "f95 -V" gave "[Errno 2] No such file or directory: 'f95'"
Running "f90 --version" gave "[Errno 2] No such file or directory: 'f90'"
Running "g95 --version" gave "[Errno 2] No such file or directory: 'g95'"
Running "g95 -V" gave "[Errno 2] No such file or directory: 'g95'"

[jpakkane]

cc1: warning: prog@exe/prog.h.gch: had text segment at different address

No idea what this could be and Google is very unhelpful. Maybe something wonky with the compiler setup (pch is not used a lot, we have come across broken pch support on many platforms).

I only have python3.5

The test suite at this time expects to find an executable python3. The code itself does not really but for example all the shebang lines in Meson are "#!/usr/bin/env python3.

These symbols are in libintl (also on OS X and some other systems).

Does it work if you change the meson.build file in src to this:

executable('intlprog', 'intlmain.c', install : true,
  dependencies : meson.get_compiler('c').find_library('intl', required : false))

The missing dep tests should start working once you install the required deps. Though for wxwidgets we also have #639.

[0-wiz-0]

Sorry for the late reply, holidays.

pre-compiled headers with gcc-5.3 on NetBSD are known to be problematic (or perhaps even broken).

I've added a symlink of python3.5 to python3. I don't think the tests should assume this, but test for it and just call the test scripts with the right interpreter.

Changing gettext's meson.build as suggested does indeed fix the test. Now I'm down to:

Total passed tests: 140
Total failed tests: 35
Total skipped tests: 31

Judging from your feedback I'll assume this to be basically working.
It'd be great if you could commit your meson.build change.

Please think about handling the non-existence of python3 better. Thank you!

[jpakkane]

It'd be great if you could commit your meson.build change.

This has been submitted to master.

Please think about handling the non-existence of python3 better. Thank you!

There are a ton of Python scripts in the wild that have #!/usr/bin/env python3 in their shebang lines. How does NetBSD handle these? Based on your description all of these should be broken (and shebang lines do not make it possible to have more than one executable name).

[0-wiz-0]

When packaging software, we override the interpreter in the shebang lines.
Some software is smart enough to go through a list of possible interpreter names (something like python3.5 python3.4 python3.3 python3 python2.7 python2 in any order) and pick the first existing one, and create proper shebang lines itself.
For test cases before installation, it's even easier, just call the scripts using the detected python interpreter.