Some tests fail on FreeBSD

[yurivict]

Some of failed tests:

Test PCE/NI_REGRESSION2/cmosInverter_regr..............FAILED[sh]
$ cat ./work/Xyce_Test/Netlists/PCE/NI_REGRESSION2/cmosInverter_regr.cir.stdout
Tecplot file contained correct number(1) of titles.
Failed to translate TECPLOT to STD
Exit code = 2

PCE/NI_REGRESSION2/cmosInverter_regr..............FAILED[sh]
$ cat ./work/Xyce_Test/Netlists/PCE/NI_REGRESSION2/cmosInverter_regr.cir.stdout
Tecplot file contained correct number(1) of titles.
Failed to translate TECPLOT to STD
Exit code = 2

Output/SENS/sens-step-tecplot.....................FAILED[sh]
$ cat ./work/Xyce_Test/Netlists/Output/SENS/sens-step-tecplot.cir.stdout
Tecplot file contained correct number(1) of titles.
Tecplot file contained correct number(1) of titles.
Tecplot file sens-step-tecplot.cir.dat contained correct number(4) of zones.
Tecplot file sens-step-tecplot.cir.SENS.dat contained correct number(4) of zones.
Tecplot file sens-step-tecplot.cir.dat contained correct number(2) of datasetauxdata's.
Tecplot file sens-step-tecplot.cir.SENS.dat contained correct number(2) of datasetauxdata's.
Tecplot file sens-step-tecplot.cir.dat contained correct number(0) of "End of Xyce(TM) Parameter Sweep" statements.
Tecplot sens-step-tecplot.cir.dat file contained correct number(1) of variables strings.
Tecplot sens-step-tecplot.cir.SENS.dat file contained correct number(1) of variables strings.
Tecplot sens-step-tecplot.cir.SENS.dat file contained correct simulation footer
retcode = 0
Failed to translate TECPLOT to STD
Failed to translate TECPLOT to STD
Exit code = 2

Output/SENS/diode.................................FAILED[sh]
$ cat ./work/Xyce_Test/Netlists/Output/SENS/diode.cir.stdout
Tecplot file contained correct number(1) of titles.
Tecplot file contained correct number(1) of zones.
Tecplot file contained correct number(2) of datasetauxdata's.
Tecplot file contained correct number(1) of variables strings.
Tecplot .SENS.dat file contained correct simulation footer
Failed to translate TECPLOT to STD
Exit code = 2

Output/HB/hb-csv..................................EXITED WITH ERROR[sh]
$ cat ./work/Xyce_Test/Netlists/Output/HB/hb-csv.cir.stdout
Exit code = 10

[tvrusso]

"Failed to translate TECPLOT to STD" happens when the "convertToPrn.py" script fails. This happens if you don't have numpy installed. Do you have numpy installed?

The hb-csv test exiting with error 10 means Xyce exited with an error, and the stdout file won't tell you why --- that file contains the output of the shell script that runs Xyce, not the Xyce run itself. Please look at the file "hb-csv.cir.out" in the work/Xyce_Test/Netlists/Output directory and tell us what it says there.

[tvrusso]

Please see the caveat in the Running the Regression Suite web page regarding prerequisites for the regression suite to pass everything. It tells you to expect failures in the Output directory if you don't have scipy and numpy.

Your hb-csv issue is likely something different, we have not seen that before.

Prerequisites

The primary test scripts that control the test suite are Perl scripts, and so you must have Perl installed on your system. One of the tests (Certification_Tests/BUG_2007/gstep) requires the Perl Digest::MD5 module. This module is present in many Perl installs on systems we’ve encountered, but we have found some that require installing an extra package to make it available.

A number of tests also use python scripts to do their work, so you must also have python installed. Many tests that use python also use the "numpy" python module, and therefore you also need to have numpy installed. A smaller number of tests in the suite requires SciPy. Perl, Python, and numpy can usually be obtained through system package managers. SciPy may be available in your system’s package manager, but on some systems (including Cygwin) it might need to be obtained from source code. If you run the test suite without having numpy or scipy, you will see about 58 test failures, most of them in the "Output" directory.

[yurivict]

With all dependencies installed and latest fixes only one test failed:

CommandLine/command_line..........................TIME LIMIT[sh] (Time: 20s = 17.77cs + 0vs)

command_line.cir.stdout and command_line.cir.stderr fles are empty.

[tvrusso]

That should be nothing to worry about. The test should be a very fast one, and has an extremely short timeout (20s). All it does is run Xyce with various command line options (like -h, -v, etc.) and check that their output matches what is expected. It runs "grep" many times, and any number of unimportant reasons can slow it down.

If your system is a little slow, or even just heavily loaded doing other things, it could fail just because of that.

It produces a number of output files other than "command_line.cir.stdout/stderr", such as capabilities.{out,err,}, count.{out,err} etc.

If all the ".err" files are empty, it just ran out of time trying to check all the files against expectations.

If you want to make absolutely sure, you can always just change the contents of the "options" file in the test's directory to something other than timelimit=20 (say, timelimit=60 or something). But I wouldn't worry about it at all if this is the only test that failed, and it was only for the time limit reason.

[yurivict]

Maybe it would make sense to increase the timeout for this testcase?

[tvrusso]

Well, it might make sense for you to increase the timeout for your use case, but it doesn't make sense for us to do it on our end.

This test is run many times a day and hundreds of times a week at Sandia on many different computers (windows, mac, and Linux), and pretty much never gets timed out with the 20 seconds limit. In fact, 20 seconds is already a generous time limit --- on our systems it never takes more than 1 or 2 seconds, and on our systems this test consistently taking 20 seconds (as opposed to taking 20 seconds on a heavily loaded system every once in a while) would be a sign of a problem that we need to fix. And the primary purpose of the regression suite is for us to make sure we don't break Xyce during development.

We probably won't be increasing that time limit because this time limit suits our needs, but you can feel free to make the port do a "sed" on Xyce_Regression/Netlists/CommandLine/options to replace "20" with something that will make it pass on every heavily loaded system it encounters. If it does this before the test cycle is run, users running the port build on slow (or heavily loaded) machines won't have to see it. And if it's your concern that FreeBSD's automated packaging processes might flag this as a broken port, then that is what you should do.

[yurivict]

Ok then. Closing.

Thanks for your help!