Fix validation failures #299

snowleopard · 2017-02-09T22:39:27Z

At the moment building GHC with Hadrian and then running build validate produces a lot of failures:

SUMMARY for test run started at 02/09/17 23:04:48 W. Europe Standard Time
 0:18:21 spent to go through
    5679 total tests, which gave rise to
   17340 test cases, of which
   11763 were skipped

      52 had missing libraries
    4783 expected passes
      94 expected failures

      24 caused framework failures
       2 unexpected passes
     618 unexpected failures
       4 unexpected stat failures

In the list of unexpected failures the majority comes from ghci, perf, printer, profiling, rts, safeHaskell and th.

Most probably cause is that the GHC we build is missing some features.

/cc: @bgamari @nomeata

The text was updated successfully, but these errors were encountered:

nomeata · 2017-02-09T23:42:10Z

Can you put the full log of the testsuite somewhere?

snowleopard · 2017-02-09T23:50:25Z

@nomeata I don't have the full log, but I attached the list of all failures to this comment.

validate-res.txt

P.S.: This particular log was generated in the default build flavour, but the results are pretty similar to the quick build flavour.

nomeata · 2017-02-10T04:29:44Z

Ah, this is on windows? How many of these are also on Linux? But without a full build log it is hard to see what’s going on. Maybe some linker error that affects Template Haskell and GHCi?

snowleopard · 2017-02-11T11:15:45Z

@nomeata Yes, I develop on Windows and test Hadrian on Linux/OSX via CI only.

Perhaps, you could try validating on Linux and share the log here?

izgzhen · 2017-04-18T08:21:45Z

On my Linux machine for commit c158014 (GHC: 65b185d):

SUMMARY for test run started at Tue Apr 18 15:55:57 2017 CST
 0:14:29 spent to go through
    5847 total tests, which gave rise to
   14389 test cases, of which
    8672 were skipped

      52 had missing libraries
    4743 expected passes
     148 expected failures

      24 caused framework failures
       0 caused framework warnings
       2 unexpected passes
     705 unexpected failures
      43 unexpected stat failures

And some log: https://gist.github.com/izgzhen/a7b8bf3bce035432f5d8fd5605136062

snowleopard · 2017-04-18T10:04:32Z

Thanks @izgzhen! This looks similar to Windows stats above.

izgzhen · 2017-09-29T05:20:16Z

A more recent result: Note that I found it is important to run validate after a default build, rather than quickest build. The later case will cause more validation failures. The Make-based system will rebuild everything using a validate flavour, which is very different from how we handle it in Hadrian.

SUMMARY for test run started at Fri Sep 29 12:57:55 2017 HKT
 0:14:21 spent to go through
    6106 total tests, which gave rise to
   23999 test cases, of which
   17951 were skipped

      43 had missing libraries
    5820 expected passes
      91 expected failures

       0 caused framework failures
       0 caused framework warnings
       0 unexpected passes
      87 unexpected failures
       7 unexpected stat failures

Unexpected failures:
   annotations/should_compile/th/annth_make.run       annth_make [bad stdout] (normal)
   annotations/should_compile/th/annth_compunits.run  annth_compunits [bad stdout] (normal)
   codeGen/should_run/T10414.run                      T10414 [exit code non-0] (threaded2)
   concurrent/should_run/conc069.run                  conc069 [bad exit code] (ghci)
   concurrent/should_run/conc069a.run                 conc069a [bad exit code] (ghci)
   concurrent/should_run/conc070.run                  conc070 [bad stdout] (ghci)
   concurrent/should_run/conc072.run                  conc072 [exit code non-0] (threaded2)
   concurrent/should_run/numsparks001.run             numsparks001 [exit code non-0] (threaded1)
   concurrent/should_run/T3429.run                    T3429 [exit code non-0] (debug_numa)
   concurrent/should_run/T5558.run                    T5558 [exit code non-0] (threaded2)
   driver/dynHelloWorld.run                           dynHelloWorld [exit code non-0] (dyn)
   dynlibs/T5373.run                                  T5373 [bad exit code] (normal)
   dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   dynlibs/T13702.run                                 T13702 [bad exit code] (normal)
   ffi/should_run/ffi014.run                          ffi014 [exit code non-0] (threaded1)
   ffi/should_run/Capi_Ctype_001.run                  Capi_Ctype_001 [bad exit code] (normal)
   ffi/should_run/Capi_Ctype_002.run                  Capi_Ctype_002 [bad exit code] (normal)
   hsc2hs/hsc2hs001.run                               hsc2hs001 [bad exit code] (normal)
   hsc2hs/hsc2hs002.run                               hsc2hs002 [bad exit code] (normal)
   hsc2hs/hsc2hs003.run                               hsc2hs003 [bad exit code] (normal)
   hsc2hs/hsc2hs004.run                               hsc2hs004 [bad exit code] (normal)
   hsc2hs/T3837.run                                   T3837 [bad exit code] (normal)
   hsc2hs/T4340.run                                   T4340 [bad exit code] (normal)
   hsc2hs/T10272.run                                  T10272 [bad exit code] (normal)
   hsc2hs/T12504.run                                  T12504 [bad exit code] (normal)
   plugins/frontend01.run                             frontend01 [bad stdout] (normal)
   pmcheck/should_compile/T11195.run                  T11195 [exit code non-0] (normal)
   printer/T13550.run                                 T13550 [bad stdout] (normal)
   printer/T13942.run                                 T13942 [bad stdout] (normal)
   printer/T13199.run                                 T13199 [bad stdout] (normal)
   profiling/should_run/T2592.run                     T2592 [bad exit code] (profasm)
   profiling/should_run/T3001-2.run                   T3001-2 [exit code non-0] (prof_hb)
   profiling/should_run/scc003.run                    scc003 [bad profile] (profasm)
   profiling/should_run/scc001.run                    scc001 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc002.run                    scc002 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc003.run                    scc003 [bad profile] (prof)
   profiling/should_run/scc003.run                    scc003 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc005.run                    scc005 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T680.run                      T680 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T2552.run                     T2552 [bad exit code] (ghci-ext-prof)
   profiling/should_run/prof-doc-fib.run              prof-doc-fib [bad exit code] (ghci-ext-prof)
   profiling/should_run/prof-doc-last.run             prof-doc-last [bad exit code] (ghci-ext-prof)
   profiling/should_run/T5559.run                     T5559 [bad exit code] (ghci-ext-prof)
   profiling/should_run/profinline001.run             profinline001 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (profasm)
   profiling/should_run/T5363.run                     T5363 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hc_hb)
   profiling/should_run/T12962.run                    T12962 [bad profile] (profasm)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hb)
   profiling/should_run/T12962.run                    T12962 [bad profile] (prof)
   profiling/should_run/T12962.run                    T12962 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hd)
   quasiquotation/T14028.run                          T14028 [bad stderr] (normal)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hy)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hr)
   rts/bug1010.run                                    bug1010 [exit code non-0] (normal)
   rts/T2047.run                                      T2047 [bad exit code] (normal)
   rts/testwsdeque.run                                testwsdeque [exit code non-0] (threaded1)
   rts/atomicxchg.run                                 atomicxchg [exit code non-0] (threaded1)
   rts/T4850.run                                      T4850 [bad exit code] (normal)
   rts/ffishutdown.run                                ffishutdown [exit code non-0] (threaded1)
   rts/T8209.run                                      T8209 [bad stderr] (ghci)
   rts/T8242.run                                      T8242 [bad stderr] (ghci)
   rts/T9078.run                                      T9078 [exit code non-0] (threaded1)
   rts/T8124.run                                      T8124 [exit code non-0] (threaded1)
   rts/linker_error1.run                              linker_error1 [bad exit code] (normal)
   rts/linker_error2.run                              linker_error2 [bad exit code] (normal)
   rts/linker_error3.run                              linker_error3 [bad exit code] (normal)
   rts/T10728.run                                     T10728 [exit code non-0] (threaded2)
   rts/T10296b.run                                    T10296b [exit code non-0] (threaded2)
   rts/numa001.run                                    numa001 [exit code non-0] (debug_numa)
   rts/T1791/T1791.run                                T1791 [bad exit code] (normal)
   rts/T5644/T5644.run                                T5644 [bad exit code] (optasm)
   simplCore/should_run/simplrun010.run               simplrun010 [bad exit code] (optasm)
   th/TH_spliceE5_prof_ext.run                        TH_spliceE5_prof_ext [bad exit code] (normal)
   th/TH_spliceE5_prof.run                            TH_spliceE5_prof [bad exit code] (normal)
   th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)
   typecheck/T13168/T13168.run                        T13168 [bad stderr] (normal)
   typecheck/should_compile/T11462.run                T11462 [exit code non-0] (normal)
   typecheck/should_compile/T11525.run                T11525 [exit code non-0] (normal)
   ../../libraries/base/tests/IO/T12010/T12010.run    T12010 [bad exit code] (threaded1)
   ../../libraries/process/tests/T3231.run            T3231 [exit code non-0] (threaded1)
   ../../libraries/process/tests/T3994.run            T3994 [exit code non-0] (threaded1)
   ../../libraries/stm/tests/stm056.run               stm056 [exit code non-0] (threaded1)
   ../../libraries/unix/tests/forkprocess01.run       forkprocess01 [exit code non-0] (threaded1_ls)
   ../../libraries/unix/tests/fdReadBuf001.run        fdReadBuf001 [bad exit code] (ghci)
   rts/T7815.run                                      T7815 [exit code non-0] (threaded1)

Unexpected stat failures:
   perf/compiler/T1969.run    T1969 [stat not good enough] (normal)
   perf/compiler/T5631.run    T5631 [stat too good] (normal)
   perf/compiler/T5837.run    T5837 [stat too good] (normal)
   perf/compiler/T9872a.run   T9872a [stat too good] (normal)
   perf/compiler/T12150.run   T12150 [stat too good] (optasm)
   perf/should_run/T5205.run  T5205 [stat too good] (normal)
   perf/should_run/T9203.run  T9203 [stat too good] (normal)

snowleopard · 2017-09-29T19:56:13Z

@izgzhen Thanks for the update! Reducing 705 unexpected failures to 87 unexpected failures simply by switching to the default build flavour is great :)

Indeed, it looks like there is a special flavour for validation.

bgamari · 2017-09-30T15:25:44Z

I suppose many of these were due to SplitSections being enabled by default which appears to break the RTS linker. I'm tracking this down in #14291.

izgzhen · 2017-10-18T02:06:23Z

The latest result with default flavour showed a regression:

     165 unexpected failures
       7 unexpected stat failures

I think I shall drop the PR #422 for now, since I still haven't find any clear way to get a better validation result yet.

chitrak7 · 2018-03-02T15:48:04Z

Results generated in ubuntu 17.10 on 2nd March 2017:
steps:

./build.sh clean
./build.sh --rebuilding hadrian
./build.sh validate -VV {I added a third argument to make command in Settings.Builders.Make "VERBOSE=1"}
Here is the result summary:
SUMMARY for test run started at Fri Mar 2 20:16:57 2018 IST
0:19:23 spent to go through
6262 total tests, which gave rise to
24521 test cases, of which
18315 were skipped

41 had missing libraries
5945 expected passes
97 expected failures
```
0 caused framework failures
0 caused framework warnings
0 unexpected passes
```
118 unexpected failures
5 unexpected stat failures

Complete Summary:
results_val2.txt
Error Generated in failures :
errorlog_2.txt

snowleopard · 2018-03-02T16:04:03Z

Thanks @chitrak7! Hope you'll be able to figure out the cause of some validation failures.

chitrak7 · 2018-03-02T16:06:24Z

@snowleopard 55 of 118 unexpected failures have come from printer/. But I seriously doubt it has something to do with my system configuration and not the build. Can you review their error log once to check this. I have attached the errorlog in the previous comment.

snowleopard · 2018-03-02T16:09:10Z

@chitrak7 I don't think I can provide any useful comments on the validation log: I haven't looked into validation errors yet, so it's a completely new territory for me.

chitrak7 · 2018-03-03T19:12:23Z

@snowleopard for a lot of failures, the reason is that some executable is not present in inplace. For example,
inplace/bin/check-api-annotations
inplace/bin/check-ppr
inplace/test spaces/runghc

I think solving this might reduce a lot of errors

snowleopard · 2018-03-03T19:41:41Z

@chitrak7 Aha, interesting! Presumably, these are only needed for the testsuite, so we didn't bother to build them before.

We need to add missing packages here:

https://github.com/snowleopard/hadrian/blob/master/src/GHC.hs#L45-L100

And also need them in the validate rule before running the testsuite. Would you like to give this a try?

chitrak7 · 2018-03-03T20:01:58Z

Yeah, would be a nice start. Would need some time and a lot of help though 😅

snowleopard · 2018-03-03T20:15:27Z

@chitrak7 Awesome, thanks. Happy to help!

chitrak7 · 2018-03-05T17:37:45Z

Results after adding missing executables:
SUMMARY for test run started at Mon Mar 5 22:30:39 2018 IST
0:20:47 spent to go through
6262 total tests, which gave rise to
24521 test cases, of which
18315 were skipped

  41 had missing libraries
6020 expected passes
  97 expected failures

   3 caused framework failures
   0 caused framework warnings
   0 unexpected passes
  45 unexpected failures
   3 unexpected stat failures

errorlog_3.txt
results_val3.txt
@snowleopard

snowleopard · 2018-03-06T15:00:36Z

@chitrak7 That's a very promising result! Can you prepare a PR?

Furthermore, it looks like the remaining 45 unexpected failures are due to 41 had missing libraries, which probably means Hadrian doesn't build some libraries needed for the testsuite.

chitrak7 · 2018-03-06T15:50:01Z

@snowleopard Currently I am trying to single out more libraries that are causing error. I have identified a couple more. I will try to bring the errors down to 20s and then send the PR. Need a couple more days.

snowleopard · 2018-03-06T15:55:30Z

@chitrak7 No need to send just one perfect PR. In fact, sending a PR to only adds missing executables would be better, because it's easier to review and understand in isolation.

snowleopard · 2019-07-17T12:31:11Z

I believe this issue is out of date, so closing.

snowleopard added bug help wanted infrastructure labels Feb 9, 2017

bgamari modified the milestones: tree-tremble, ghc-quake May 10, 2017

izgzhen mentioned this issue Sep 30, 2017

Add validate flavour #422

Closed

snowleopard mentioned this issue Dec 4, 2017

Rethink the -Werror decision. #485

Closed

snowleopard closed this as completed Jul 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix validation failures #299

Fix validation failures #299

snowleopard commented Feb 9, 2017

nomeata commented Feb 9, 2017

snowleopard commented Feb 9, 2017 •

edited

nomeata commented Feb 10, 2017

snowleopard commented Feb 11, 2017

izgzhen commented Apr 18, 2017 •

edited

snowleopard commented Apr 18, 2017

izgzhen commented Sep 29, 2017

snowleopard commented Sep 29, 2017 •

edited

bgamari commented Sep 30, 2017

izgzhen commented Oct 18, 2017

chitrak7 commented Mar 2, 2018

snowleopard commented Mar 2, 2018

chitrak7 commented Mar 2, 2018 •

edited

snowleopard commented Mar 2, 2018 •

edited

chitrak7 commented Mar 3, 2018

snowleopard commented Mar 3, 2018

chitrak7 commented Mar 3, 2018

snowleopard commented Mar 3, 2018

chitrak7 commented Mar 5, 2018 •

edited

snowleopard commented Mar 6, 2018 •

edited

chitrak7 commented Mar 6, 2018

snowleopard commented Mar 6, 2018

snowleopard commented Jul 17, 2019

Fix validation failures #299

Fix validation failures #299

Comments

snowleopard commented Feb 9, 2017

nomeata commented Feb 9, 2017

snowleopard commented Feb 9, 2017 • edited

nomeata commented Feb 10, 2017

snowleopard commented Feb 11, 2017

izgzhen commented Apr 18, 2017 • edited

snowleopard commented Apr 18, 2017

izgzhen commented Sep 29, 2017

snowleopard commented Sep 29, 2017 • edited

bgamari commented Sep 30, 2017

izgzhen commented Oct 18, 2017

chitrak7 commented Mar 2, 2018

snowleopard commented Mar 2, 2018

chitrak7 commented Mar 2, 2018 • edited

snowleopard commented Mar 2, 2018 • edited

chitrak7 commented Mar 3, 2018

snowleopard commented Mar 3, 2018

chitrak7 commented Mar 3, 2018

snowleopard commented Mar 3, 2018

chitrak7 commented Mar 5, 2018 • edited

snowleopard commented Mar 6, 2018 • edited

chitrak7 commented Mar 6, 2018

snowleopard commented Mar 6, 2018

snowleopard commented Jul 17, 2019

snowleopard commented Feb 9, 2017 •

edited

izgzhen commented Apr 18, 2017 •

edited

snowleopard commented Sep 29, 2017 •

edited

chitrak7 commented Mar 2, 2018 •

edited

snowleopard commented Mar 2, 2018 •

edited

chitrak7 commented Mar 5, 2018 •

edited

snowleopard commented Mar 6, 2018 •

edited