Skip to content
This repository has been archived by the owner on Jul 1, 2021. It is now read-only.

Most regression tests fail on FreeBSD #211

Closed
yurivict opened this issue Jan 4, 2020 · 16 comments · Fixed by #222
Closed

Most regression tests fail on FreeBSD #211

yurivict opened this issue Jan 4, 2020 · 16 comments · Fixed by #222

Comments

@yurivict
Copy link
Contributor

yurivict commented Jan 4, 2020

[==========] Running 294 tests from 16 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from array
[ RUN      ] array.run_2
/usr/ports/cad/cascade-compiler/work/cascade-3f0c4988aacc8bde4b2661c6666c44eeb3024573/test/harness.cc:83: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_2 (6 ms)
[ RUN      ] array.run_3
/usr/ports/cad/cascade-compiler/work/cascade-3f0c4988aacc8bde4b2661c6666c44eeb3024573/test/harness.cc:83: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_3 (6 ms)
[ RUN      ] array.run_4
/usr/ports/cad/cascade-compiler/work/cascade-3f0c4988aacc8bde4b2661c6666c44eeb3024573/test/harness.cc:83: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_4 (6 ms) 

Some succeed:

[       OK ] type_check.fail_array_1 (5 ms)
[ RUN      ] type_check.fail_array_2
[       OK ] type_check.fail_array_2 (5 ms)
[ RUN      ] type_check.fail_array_3
[       OK ] type_check.fail_array_3 (6 ms)
[ RUN      ] type_check.fail_array_4
[       OK ] type_check.fail_array_4 (5 ms)
[ RUN      ] type_check.fail_array_5
[       OK ] type_check.fail_array_5 (5 ms)
[ RUN      ] type_check.fail_array_6
[       OK ] type_check.fail_array_6 (5 ms)
@eschkufz
Copy link
Contributor

eschkufz commented Jan 4, 2020

Well... this isn't saying much... those type_check.fail_xxx tests only pass because they're expected to fail.

I added some code to the last pull request to print out cascade's error stream whenever a test fails. Can you try rerunning your regression tests to see what the errors look like?

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

Now the log looks like this:

[==========] Running 294 tests from 16 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from array
[ RUN      ] array.run_2
Parse Error:
  > Unable to locate file share/cascade/march/regression/minimal.v
  > In final line of user input:
    syntax error, unexpected <unparseable>
/usr/ports/cad/cascade-compiler/work/cascade-5a27bf54d8b9d6e708192d59d99623f428a603b4/test/harness.cc:87: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_2 (6 ms)
[ RUN      ] array.run_3
Parse Error:
  > Unable to locate file share/cascade/march/regression/minimal.v
  > In final line of user input:
    syntax error, unexpected <unparseable>
/usr/ports/cad/cascade-compiler/work/cascade-5a27bf54d8b9d6e708192d59d99623f428a603b4/test/harness.cc:87: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_3 (7 ms)
[ RUN      ] array.run_4
Parse Error:
  > Unable to locate file share/cascade/march/regression/minimal.v
  > In final line of user input:
    syntax error, unexpected <unparseable>
/usr/ports/cad/cascade-compiler/work/cascade-5a27bf54d8b9d6e708192d59d99623f428a603b4/test/harness.cc:87: Failure
Value of: c.bad()
  Actual: true
Expected: false
[  FAILED  ] array.run_4 (6 ms)

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

I think your file search is still broken.

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

I used out-of-source build in /usr/ports/cad/cascade-compiler/work/.build, executable run_regression is located there, but installation was done into /usr/ports/cad/cascade-compiler/work/stage/usr/local/...

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

It looks for share/cascade/test/regression/bitcoin/bitcoin_2.v, didn't you just remove it from the installation? -)

@eschkufz
Copy link
Contributor

eschkufz commented Jan 4, 2020

Yeah, I just removed the test and benchmark files from the installation. The regression tests will only work if you run them the cascade source directory since they'll all be looking for the share/ directory relative to the current working directory.

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

You need to make regression tests to look for these files in the build directory.

@eschkufz
Copy link
Contributor

eschkufz commented Jan 4, 2020

Gotcha. Let me see what I can do about that.

@eschkufz
Copy link
Contributor

eschkufz commented Jan 4, 2020

Try the most recent commit to issue-211. I updated cmake to copy the share/cascade/ directory to the build directory. If you run make regression from inside your build directory it should work, regardless of where your build directory is relative to your source directory.

@yurivict
Copy link
Contributor Author

yurivict commented Jan 4, 2020

Now regression target test fails when run as a target:

    Start 1: regression
1/1 Test #1: regression .......................***Timeout 1500.02 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 1500.03 sec

The following tests FAILED:
	  1 - regression (Timeout)

but ./test/run_regression succeeds:

<...skipped...>
[----------] 5 tests from verilator64
[ RUN      ] verilator64.array
[       OK ] verilator64.array (1664 ms)
[ RUN      ] verilator64.bitcoin
[       OK ] verilator64.bitcoin (24736 ms)
[ RUN      ] verilator64.mips32
[       OK ] verilator64.mips32 (1788 ms)
[ RUN      ] verilator64.nw
[       OK ] verilator64.nw (137 ms)
[ RUN      ] verilator64.regex
[       OK ] verilator64.regex (914 ms)
[----------] 5 tests from verilator64 (29239 ms total)

[----------] Global test environment tear-down
[==========] 294 tests from 16 test suites ran. (184993 ms total)
[  PASSED  ] 294 tests.

@eschkufz
Copy link
Contributor

eschkufz commented Jan 5, 2020

That sounds like progress. Can you show me the command lines you're using that get you from a fresh checkout to this error? I want to try duplicating your directory structure to see if I can reproduce on my end.

@yurivict
Copy link
Contributor Author

yurivict commented Jan 5, 2020

configure is done with

/bin/mkdir -p /usr/ports/cad/cascade-compiler/work/.build
cd /usr/ports/cad/cascade-compiler/work/.build; /usr/bin/env ac_cv_path_PERL=/usr/local/bin/perl ac_cv_path_PERL_PATH=/usr/local/bin/perl  PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/usr/ports/cad/cascade-compiler/work  XDG_CONFIG_HOME=/usr/ports/cad/cascade-compiler/work  HOME=/usr/ports/cad/cascade-compiler/work PATH=/usr/ports/cad/cascade-compiler/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin SHELL=/bin/sh CONFIG_SHELL=/bin/sh CMAKE_PREFIX_PATH="/usr/local" /usr/local/bin/cmake -DCMAKE_C_COMPILER:STRING="cc"  -DCMAKE_CXX_COMPILER:STRING="c++"  -DCMAKE_C_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_C_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_C_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_CXX_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  -DCMAKE_CXX_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  -DCMAKE_CXX_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  -DCMAKE_EXE_LINKER_FLAGS:STRING=" -Wl,-rpath=/usr/local/lib -fstack-protector-strong -L/usr/local/lib "  -DCMAKE_MODULE_LINKER_FLAGS:STRING=" -Wl,-rpath=/usr/local/lib -fstack-protector-strong -L/usr/local/lib "  -DCMAKE_SHARED_LINKER_FLAGS:STRING=" -Wl,-rpath=/usr/local/lib -fstack-protector-strong -L/usr/local/lib "  -DCMAKE_INSTALL_PREFIX:PATH="/usr/local"  -DCMAKE_BUILD_TYPE:STRING="Release"  -DTHREADS_HAVE_PTHREAD_ARG:BOOL=YES  -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=YES  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DBUILD_TESTING:BOOL=OFF -GNinja /usr/ports/cad/cascade-compiler/work/cascade-25e1050ab41c242c77014d7436998611e0b3fb82

Build is done with

(cd /usr/ports/cad/cascade-compiler/work/.build; if ! /usr/bin/env FREEBSD_WRKSRC=/usr/ports/cad/cascade-compiler/work/cascade-25e1050ab41c242c77014d7436998611e0b3fb82 FREEBSD_LOCALBASE=/usr/local PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/usr/ports/cad/cascade-compiler/work  XDG_CONFIG_HOME=/usr/ports/cad/cascade-compiler/work  HOME=/usr/ports/cad/cascade-compiler/work PATH=/usr/ports/cad/cascade-compiler/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES DESTDIR=/usr/ports/cad/cascade-compiler/work/stage PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer -isystem /usr/local/include"  LDFLAGS=" -Wl,-rpath=/usr/local/lib -fstack-protector-strong -L/usr/local/lib " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" ninja   -j8 -v all; then  if [ -n "Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer." ] ; then  echo "===> Compilation failed unexpectedly.";  (echo "Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer.") | /usr/bin/fmt 75 79 ;  fi;  false;  fi)

test is done with

(cd /usr/ports/devel/benchmark/work/.build; if ! /usr/bin/env XDG_DATA_HOME=/usr/ports/devel/benchmark/work  XDG_CONFIG_HOME=/usr/ports/devel/benchmark/work  HOME=/usr/ports/devel/benchmark/work PATH=/usr/ports/devel/benchmark/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES DESTDIR=/usr/ports/devel/benchmark/work/stage PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer"  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer  -Wno-thread-safety-analysis -Wno-sign-compare "  MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" ninja   -j8 -v all; then  if [ -n "Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer." ] ; then  echo "===> Compilation failed unexpectedly.";  (echo "Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to the maintainer.") | /usr/bin/fmt 75 79 ;  fi;  false;  fi)

Sorry, these commands are auto-generated by the framework, and are very long.

@eschkufz
Copy link
Contributor

eschkufz commented Jan 5, 2020 via email

@eschkufz
Copy link
Contributor

eschkufz commented Jan 5, 2020

Give the most recent commit to issue-211 a shot. Fingers crossed that does the trick.

@yurivict
Copy link
Contributor Author

yurivict commented Jan 5, 2020

Yes, tests pass now in issue-211.

@eschkufz
Copy link
Contributor

eschkufz commented Jan 5, 2020 via email

@eschkufz eschkufz mentioned this issue Jan 5, 2020
2 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants