New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Black magic in Test::YAML #3

Closed
eserte opened this Issue Aug 24, 2016 · 15 comments

Comments

Projects
None yet
3 participants
@eserte
Copy link

eserte commented Aug 24, 2016

It is not possible to just "use" Test::YAML without getting some warnings, errors, and the TAP footer:

$ perl5.18.4 -e 'use Test::YAML'                  
readline() on unopened filehandle DATA at /usr/perl5.18.4p/lib/site_perl/5.18.4/Test/Base.pm line 519.
Use of uninitialized value $spec in substitution (s///) at /usr/perl5.18.4p/lib/site_perl/5.18.4/Test/Base.pm line 415.
Use of uninitialized value $spec in pattern match (m//) at /usr/perl5.18.4p/lib/site_perl/5.18.4/Test/Base.pm line 427.
Can't determine section names at -e line 0.
END failed--call queue aborted.
1..0

This is especially problematic in conjunction with CPAN::Reporter. At the end of a test run, CPAN::Reporter is essentially doing this:

$ echo "Test::YAML 0" | perl5.18.4 /usr/perl5.18.4p/lib/site_perl/5.18.4/CPAN/Reporter/PrereqCheck.pm
Test::YAML 1 1.06
Can't determine section names at /usr/perl5.18.4p/lib/site_perl/5.18.4/CPAN/Reporter/PrereqCheck.pm line 0.
END failed--call queue aborted, <DATA> line 1.
1..0

... and the same errors and the TAP footer are visible here.

Within a CPAN.pm session while building YAML.pm it looks like this:

...
t/rt-90593.t .............. ok
t/svk.t ................... ok
t/test.t .................. ok
All tests successful.
Files=45, Tests=480, 14 wallclock secs ( 0.06 usr  0.21 sys +  3.30 cusr  0.77 csys =  4.34 CPU)
Result: PASS
(/usr/bin/make test exited with 0)
Error parsing output from CPAN::Reporter::PrereqCheck:
1..0CPAN::Reporter: Test result is 'pass', All tests successful.
CPAN::Reporter: preparing a CPAN Testers report for YAML-1.18
...

Maybe Test::YAML can find out that it's not really used within a test script (e.g. if there's no __DATA__ section) and be quiet if this is the case.

@jkeenan

This comment has been minimized.

Copy link

jkeenan commented Mar 30, 2018

I would like to second Slaven's request for a correction of this problem with Test-YAML. Here's the output I got attempting to install Test-YAML on FreeBSD-11.0 against perl-5.27.10 using cpanm.

{
  author => "INGY",
  dist => "Test-YAML",
  distname => "Test-YAML-1.06",
  distversion => 1.06,
  grade => "UNKNOWN",
  prereqs => undef,
  test_output => [
    "Building and testing Test-YAML-1.06",
    "cp lib/Test/YAML.pm blib/lib/Test/YAML.pm",
    "cp lib/Test/YAML.pod blib/lib/Test/YAML.pod",
    "cp bin/test-yaml blib/script/test-yaml",
    "\"/usr/home/jkeenan/prep528/testing/perl-5.27.10/bin/perl\" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/test-yaml",
    "PERL_DL_NONLAZY=1 \"/usr/home/jkeenan/prep528/testing/perl-5.27.10/bin/perl\" \"-MExtUtils::Command::MM\" \"-MTest::Harness\" \"-e\" \"undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')\" t/*.t",
    "t/release-pod-syntax.t .. skipped: these tests are for release candidate testing",
    "Files=1, Tests=0,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.00 csys =  0.08 CPU)",
    "Result: NOTESTS",
  ],
  via => "App::cpanminus::reporter 0.17 (1.7043)",
}

This report was generated in the course of an automated testing program designed to assess the impact of Perl 5 blead on the top 3000 CPAN distributions in the "CPAN River." A NOTESTS result from the test harness leads to a grade of UNKNOWN. That means Test-YAML does not get installed and -- more importantly from the perspective of the CPAN river -- none of its dependencies (including YAML.pm itself) get tested either.

Consequently, we are unable to currently assess how all the YAML::* modules on CPAN will fare when perl-5.28.0 is released within the next two months.

Please let us know if we can be of assistance in addressing this problem.

Thank you very much.
Jim Keenan

Since the YAML library itself has a dependency

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Mar 30, 2018

So there are actually two problems.
I think the problem @eserte mentions is already in Test::Base, so I get the same error with perl -wE'use Test::Base'. I couldn't find out how to fix this. Checking with Scalar::Util::openhandle(*DATA) will only fix one warning.

The problem that there are no tests could be fixed with a simple use test, but that won't work because of the first problem.

@ingydotnet

@jkeenan

This comment has been minimized.

Copy link

jkeenan commented Mar 30, 2018

Using the same automated testing application, I got PASS for Test-Base on FreeBSD-11.0 with perl-5.27.10 and on Ubuntu Linux 16.04 with perl-5.27.9.

{
  author => "INGY",
  dist => "Test-Base",
  distname => "Test-Base-0.88",
  distversion => 0.88,
  grade => "PASS",
  prereqs => undef,
  test_output => [
    "Building and testing Test-Base-0.88",
    "cp lib/Test/Base/Filter.pm blib/lib/Test/Base/Filter.pm",
    "cp lib/Test/Base.pm blib/lib/Test/Base.pm",
    "cp lib/Test/Base/Filter.pod blib/lib/Test/Base/Filter.pod",
    "cp lib/Test/Base.pod blib/lib/Test/Base.pod",
    "PERL_DL_NONLAZY=1 \"/home/jkeenan/var/ytad/testing/perl-5.27.9/bin/perl\" \"-MExtUtils::Command::MM\" \"-MTest::Harness\" \"-e\" \"undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')\" t/*.t",
    "t/000-require-modules.t .. ok",
    "t/append.t ............... ok",
    "t/arguments.t ............ ok",
    "t/array.t ................ ok",
    "t/autoload.t ............. ok",
    "t/base64.t ............... ok",
    "t/blocks-scalar.t ........ ok",
    "t/blocks_grep.t .......... ok",
    "t/chomp.t ................ ok",
    "t/chop.t ................. ok",
    "t/compact.t .............. ok",
    "t/compile.t .............. ok",
    "t/deep.t ................. ok",
    "t/delimiters.t ........... ok",
    "t/description.t .......... ok",
    "t/diff_is.t .............. ok",
    "t/dumper.t ............... ok",
    "t/embed_perl.t ........... ok",
    "t/escape.t ............... ok",
    "t/eval.t ................. ok",
    "t/eval_all.t ............. ok",
    "t/eval_stderr.t .......... ok",
    "t/eval_stdout.t .......... ok",
    "t/expected-zero.t ........ ok",
    "t/export.t ............... ok",
    "t/exported_func.t ........ ok",
    "t/filter_arguments.t ..... ok",
    "t/filter_delay.t ......... ok",
    "t/filter_functions.t ..... ok",
    "t/filters-append.t ....... ok",
    "t/filters.t .............. ok",
    "t/filters_map.t .......... ok",
    "t/first_block.t .......... ok",
    "t/flatten.t .............. ok",
    "t/get_url.t .............. skipped: Need to figure out network testing",
    "t/hash.t ................. ok",
    "t/head.t ................. ok",
    "t/internals.t ............ ok",
    "t/is.t ................... ok",
    "t/jit-run.t .............. ok",
    "t/join-deep.t ............ ok",
    "t/join.t ................. ok",
    "t/last.t ................. ok",
    "t/late.t ................. ok",
    "t/lazy-filters.t ......... ok",
    "t/lines.t ................ ok",
    "t/list.t ................. ok",
    "# Filters returning numbers are supposed to do munging \$_: your filter 'main::mike5' apparently doesn't.",
    "t/main_filters.t ......... ok",
    "t/multi-level-inherit.t .. ok",
    "t/name.t ................. ok",
    "t/next.t ................. ok",
    "t/no_diff.t .............. ok",
    "t/no_plan.t .............. ok",
    "t/normalize.t ............ ok",
    "t/only-with-implicit.t ... ok",
    "t/only.t ................. ok",
    "t/oo.t ................... ok",
    "t/oo_run.t ............... ok",
    "t/parentheses.t .......... ok",
    "t/prepend.t .............. ok",
    "t/preserve-order.t ....... ok",
    "t/prototypes.t ........... ok",
    "t/quick-plan.t ........... ok",
    "t/quick_test.t ........... ok",
    "t/read_file.t ............ ok",
    "t/regexp.t ............... ok",
    "t/release-pod-syntax.t ... skipped: these tests are for release candidate testing",
    "t/repeated-filters.t ..... ok",
    "t/require.t .............. ok",
    "t/reserved_names.t ....... ok",
    "t/reverse-deep.t ......... ok",
    "t/reverse.t .............. ok",
    "t/run-args.t ............. ok",
    "t/run_compare.t .......... ok",
    "t/run_is.t ............... ok",
    "t/run_is_deep.t .......... ok",
    "t/run_is_deeply.t ........ ok",
    "t/run_like.t ............. ok",
    "t/run_unlike.t ........... ok",
    "t/simple.t ............... ok",
    "t/skip.t ................. ok",
    "t/slice.t ................ ok",
    "t/sort-deep.t ............ ok",
    "t/sort.t ................. ok",
    "t/spec_file.t ............ ok",
    "t/spec_string.t .......... ok",
    "t/split-deep.t ........... ok",
    "t/split-regexp.t ......... ok",
    "t/split.t ................ ok",
    "t/strict-warnings.t ...... ok",
    "t/strict.t ............... ok",
    "t/subclass-autoclass.t ... ok",
    "t/subclass-import.t ...... ok",
    "t/subclass.t ............. ok",
    "t/subclass_late.t ........ ok",
    "t/tail.t ................. ok",
    "t/tie_output.t ........... ok",
    "t/trim.t ................. ok",
    "t/unchomp.t .............. ok",
    "t/undef.t ................ ok",
    "t/use-test-more.t ........ ok",
    "t/write_file.t ........... ok",
    "t/xxx.t .................. skipped: Requires YAML",
    "t/yaml.t ................. skipped: Requires YAML",
    "t/zero-blocks.t .......... ok",
    "All tests successful.",
    "Files=105, Tests=430, 10 wallclock secs ( 0.27 usr  0.11 sys +  8.54 cusr  1.14 csys = 10.06 CPU)",
    "Result: PASS",
  ],
  via => "App::cpanminus::reporter 0.17 (1.7043)",
}

I therefore think the problem is specific to Test-YAML.

@ingydotnet

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Mar 30, 2018

@jkeenan yes, my point was that the problem @eserte mentions happens already in Test::Base.

The problem that you are seeing is because Test::YAML does not have any tests (except one that doesn't run by default).
That could be solved by adding a simple test that use Test::YAML succeeds, but that will fail because of the first problem.

perlpunk added a commit to perlpunk/test-base-pm that referenced this issue Apr 18, 2018

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Apr 18, 2018

I created ingydotnet/test-base-pm#21 to fix the problem reported by @eserte
greetings from the perl toolchain summit

ingydotnet added a commit to ingydotnet/test-base-pm that referenced this issue Apr 19, 2018

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Apr 19, 2018

@eserte Test::Base 0.89 has been released, can you check Test::YAML with the new version?

perlpunk added a commit to perlpunk/test-yaml-pm that referenced this issue May 6, 2018

Add a test that loads the module
Currently there are no tests at all, newe cpan clients will interpret this
as failed.

See ingydotnet#3
@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented May 6, 2018

@jkeenan I created #4 to fix the problem of no tests

perlpunk added a commit to perlpunk/test-yaml-pm that referenced this issue May 6, 2018

Add a test that loads the module
Currently there are no tests at all, newe cpan clients will interpret this
as failed.

See ingydotnet#3
@jkeenan

This comment has been minimized.

Copy link

jkeenan commented May 6, 2018

While I think this work, I cannot really test it out in situ until it gets released to CPAN. That's because I need to run it within the context of a program which is using cpanm built against a specific Perl development release to install large numbers of CPAN modules in dependency order.

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented May 10, 2018

@jkeenan the current status on this is:
@ingydotnet tried to give me COMAINT on Test::YAML so I can release the fix.
However, the comaint script that comes with App::PAUSE::Comaint doesn't work anymore, probably because of recent changes to pause.perl.org. So I don't have COMAINT and am blocked on this.
miyagawa/App-PAUSE-Comaint#9

@jkeenan

This comment has been minimized.

Copy link

jkeenan commented May 13, 2018

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented May 13, 2018

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Jun 16, 2018

Finally I have commit and PAUSE comaint.
Merged #4, released Test-YAML-1.07.tar.gz

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Jun 16, 2018

cpantesters looks ok, closing

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Nov 3, 2018

Seems this is still not fixed
ingydotnet/yaml-pm#210

The problem is that Test::Base reads from *main::DATA which is the pod section of CPAN::Reporter::PrereqCheck in this case.

@perlpunk

This comment has been minimized.

Copy link
Collaborator

perlpunk commented Nov 3, 2018

I created ingydotnet/test-base-pm#22 as a replacement for this issue.

@perlpunk perlpunk closed this Nov 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment