Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Test::More doesn't prepare test env for script's children #390

Closed
Astara opened this Issue · 14 comments

5 participants

@Astara

When run under a test harness, I'd expect it to set the ENV vars needed for the test and it's children to run properly, but this isn't the case.

I have a test script "P.t", that Test::More uses to test P.pm. P.t calls P.pm as a program multiple times to get test output and compares the output against "correct responses".

Example of the problem P.pm "used" module "mem". The prereqs were loaded, and the script detected the presence of the mem in the @INC path -- HOWEVER, the @INC path doesn't represent what is in the environment in the PERL5LIB pat. So the child gets a complete different include path than the script program sees. As a result, when it tries to include pre-req modules, it fails.

Test::More needs to set the environment with any options changed in PATH and INCLUDE so they are propagated to children in a test program. Without that, the child runs in some "random environment" that causes unexpected and cryptic failures.... like:

PERL_DL_NONLAZY=1 /home/sand/src/perl/repoperls/installed-perls/perl/v5.12.5/9980/bin/perl "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/.t
Can't locate mem.pm in @INC (you may need to install the mem module) (@INC contains: [...]:.) at lib/P.pm line 150.
BEGIN failed--compilation aborted at lib/P.pm line 150.

and

------/PREREQUISITES/------
Prerequisite modules loaded:
requires:
Module Need Have

------------------- ------ --------
mem v0.3.2 0.3.2

Test::Simple 0.44 1.001002

build_requires:
Module Need Have

------------------- ------ --------

mem v0.3.2 0.3.2

Where mem is not found, but is theoretically loaded.

@Leont

It's not Test::More's task to set any environmental variables. Test::Harness sometimes does, but in this case I'd suspect it's your shell script that's being wrong. The command perl isn't guaranteed to be the correct perl installation, and overriding $PERL5LIB that way is also asking for trouble. And exec isn't being passed argument correctly either.

Not to mention the question why does that shell script exist in the first place?

@Astara

Ah... my mistake, I thought Test::More was a test harness. I'm used to test harness's that set up the test env, run a test script that runs a program with inputs and checks the outputs. But the the script just takes care of running the test, while a harness takes care of setting up the env... so, sorry, I thought Test::More was supposed to be a test harness.

The shell script is the test-harness that sets up the environment so the tests can run correctly. AT least thats the idea, as you say -- since you've said that it's not Test::More's responsibility. Something has to set it before perl is run, and if it isn't the harness, is a script. That explain how it is?

@karenetheridge
Collaborator

Something has to set it before perl is run, and if it isn't the harness, is a script.

Your test (the .t file) itself should be responsible for setting up the environment needed for any subsequent tests to properly run.

@Astara
@karenetheridge
Collaborator

Your test (the .t file) itself should be responsible

It is not able to, according to the documentation in Test::More.

I didn't say Test::More, I said your test. Your test is perl and can manipulate the environment. If you intend to call out to an external script, you should set up whatever preconditions that script requires.

All that Test::More does is provide a mechanism for comparing actual results against expected ones. It knows nothing about the code being tested or how it is supposed to work. That's on you, the test writer.

I would strongly recommend re-reading https://metacpan.org/pod/Test::Tutorial.

@Astara
@karenetheridge
Collaborator

"Your test" is your .t file.

If you invoke a separate process and want to test its output, you need to first of all invoke the process properly (that's up to you to control), and then capture stdout/stderr into a string. Then you can use Test::More to compare the actual output from what you expected.

None of this needs to be tied to a Test::* module - you are free to invoke your process any way you like, and capture its streams any way you like. Capture::Tiny and Test::Output are two popular approaches here, but there are more as well.

It's Test::More that can't read P.pm correctly.

That statement makes absolutely no sense. It's not trying to read anything; all it does (assuming we're talking about the is function) is compare two strings for equality, and emit the proper TAP output indicating the result.

You seem to be very confused about the differences between a module and a program, how to invoke separate programs, how to capture output, and what the Test::Builder framework actually does.

For that matter, I see no reason why you need to invoke separate processes at all to test a module that seems to be all about turning data structures into strings. Why not do everything in the single process?

@Astara
@shadowcat-mst

A sane solution to the underlying problem would be to take the following code in P.pm -

package main;

use 5.8.0;
use utf8;

unless ((caller 0)[0]) {
$_=do{ $/=undef, <P::DATA>};
    close P::DATA;
    our @globals;
    eval $_;
die "self-test failed: $@" if $@;
    1;
} else {
    close P::DATA;
}

do the slurp of into a variable no matter what happens, and do expose the script running thing as a subroutine, maybe run_as_script.

Then you can write

unless ((caller 0)[0]) { run_as_script() }

to preserve the "run as a script" behaviour but you can call P::run_as_script() directly from your test code.

At which point you can throw out the subprocess overcomplication, delete that shell script, and worry about how to spawn perl processes from test files later when you've got a genuine case where you need to understand that.

THis still isn't a Test::More bug but good luck refactoring P.pm so it can be tested sensibly!

@Astara
@karenetheridge
Collaborator
@shadowcat-mst

"I understand what you are saying, BUT ... "

I tried to give you some hints while explaining this isn't a Test::More bug

So long as you're now clear this isn't, @schwern can close it, and you can redirect your support request to another, more appropriate venue. You might want to try perlmonks; they're usually quite sympathetic to newbies trying to learn the basics and certainly more so than I'm capable of.

@Astara
@rjbs
Owner

Tests are generally run by Test::Harness or TAP::Harness, neither of which is part of Test::More.

I'm closing this ticket. TAP-Harness has its own issues list.

@rjbs rjbs closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.