RunExtraTests runs all files under xt/, including data files.

[van-de-bugger]

I have xt/ directory, it contains few tests (with .t suffix) and few non-test, non-Perl non-scripts, but supporting data for tests. Namely, I have perlcritic.ini (a config file for Test::Perl::Critic) and aspell.en.pws (custom dictionary for Test::PodSpelling). dzil test and dzil xtest works as expected: the first command runs all the tests under t/, the second one runs all the tests under xt/. Adding CheckExtraTests to my dist.ini also works as expected: dzil release runs t/ tests first (since I also use TestRelease), then runs xt/ tests (and builds the distro twice; but it is documented).

Documentation recommends using RunExtraTests to avoid double build. Unfortunately, RunExtraTests does not work for me: dzil test runs all the files under t/ and xt/, including non-Perl, non-.t files. Obviously, xt/perlcritic.ini and xt/aspell.en.pws fail. dzil xtest works as before, it runs only tests, xt/*.t, and ignores non-test files.

[dagolden]

@karenetheridge any idea what's going on with this?

[karenetheridge]

I'd want to see the output of each of these commands, and an ls -lR of the t and xt directories. It could be a Test::Harness or App::Prove problem, or something strange is up with the files themselves.

[van-de-bugger]

Here is the reproducer:

; dist.ini
name                = Assa
abstract            = Abstarct
version             = 0.001
[GenerateFile/Assa.pm]
    filename = lib/Assa.pm
    content  = package Assa;
    content  = 1;
[GenerateFile/perlcritic.ini]
    filename = xt/perlcritic.ini
    content  = severity = 5
    content  = verbose = 3
[Test::Compile]
[Test::EOL]
[Test::NoTabs]
[MetaYAML]
[ModuleBuildTiny]
;~ [RunExtraTests]

Output:

$ dzil test
[DZ] building distribution under .build/RV5CaML1na for installation
[DZ] beginning to build Assa
[DZ] writing Assa in .build/RV5CaML1na
Creating new 'Build' script for 'Assa' version '0.001'
cp lib/Assa.pm blib/lib/Assa.pm
t/00-compile.t .. ok   
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.05 cusr  0.01 csys =  0.09 CPU)
Result: PASS
[DZ] all's well; removing .build/RV5CaML1na

$ dzil xtest
[DZ] building distribution under .build/Br7DTOAK_O for installation
[DZ] beginning to build Assa
[DZ] writing Assa in .build/Br7DTOAK_O
Creating new 'Build' script for 'Assa' version '0.001'
cp lib/Assa.pm blib/lib/Assa.pm
xt/author/eol.t ...... ok   
xt/author/no-tabs.t .. ok   
All tests successful.
Files=2, Tests=4,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.11 cusr  0.02 csys =  0.17 CPU)
Result: PASS
all's well; removing .build/Br7DTOAK_O

Everything is ok. Now uncomment the last line of dist.ini and run dzil xtest:

$ dzil xtest
[DZ] building distribution under .build/VjX8qDdfSg for installation
[DZ] beginning to build Assa
[DZ] writing Assa in .build/VjX8qDdfSg
Creating new 'Build' script for 'Assa' version '0.001'
cp lib/Assa.pm blib/lib/Assa.pm
xt/author/eol.t ...... ok   
xt/author/no-tabs.t .. ok   
All tests successful.
Files=2, Tests=4,  0 wallclock secs ( 0.04 usr  0.01 sys +  0.09 cusr  0.02 csys =  0.16 CPU)
Result: PASS
all's well; removing .build/VjX8qDdfSg

And dzil test:

$ dzil test
[DZ] building distribution under .build/Ewq8YpdQ3h for installation
[DZ] beginning to build Assa
[DZ] writing Assa in .build/Ewq8YpdQ3h
Creating new 'Build' script for 'Assa' version '0.001'
cp lib/Assa.pm blib/lib/Assa.pm
t/00-compile.t .. ok   
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.06 cusr  0.01 csys =  0.11 CPU)
Result: PASS
xt/author/eol.t ...... ok   
xt/author/no-tabs.t .. ok   
xt/perlcritic.ini .... Semicolon seems to be missing at xt/perlcritic.ini line 1.
Can't modify constant item in scalar assignment at xt/perlcritic.ini line 2, near "verbose "
syntax error at xt/perlcritic.ini line 2, near "verbose "
Execution of xt/perlcritic.ini aborted due to compilation errors.
xt/perlcritic.ini .... Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 

Test Summary Report
-------------------
xt/perlcritic.ini  (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=3, Tests=4,  0 wallclock secs ( 0.03 usr  0.02 sys +  0.11 cusr  0.02 csys =  0.18 CPU)
Result: FAIL
[RunExtraTests] Fatal errors in xt tests
[RunExtraTests] Fatal errors in xt tests at .../lib/perl5/x86_64-linux-thread-multi/Moose/Meta/Method/Delegation.pm line 110.

You see, it tries to execute xt/perlcritic.ini as test, and this "test" fails.

[dagolden]

I think the best thing is to paste them on gist.github.com and post the links here.

[dagolden]

Or upload a tarball to Dropbox and post that link.

[van-de-bugger]

Ok, thanks. I'll do it next time. Today I already updated the original post with small one-file reproducer. It does not require external files and can be easily copy-pasted from the post.

[karenetheridge]

I found the problem; will send a PR.