New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix validation failures #299

Open
snowleopard opened this Issue Feb 9, 2017 · 22 comments

Comments

Projects
None yet
5 participants
@snowleopard
Owner

snowleopard commented Feb 9, 2017

At the moment building GHC with Hadrian and then running build validate produces a lot of failures:

SUMMARY for test run started at 02/09/17 23:04:48 W. Europe Standard Time
 0:18:21 spent to go through
    5679 total tests, which gave rise to
   17340 test cases, of which
   11763 were skipped

      52 had missing libraries
    4783 expected passes
      94 expected failures

      24 caused framework failures
       2 unexpected passes
     618 unexpected failures
       4 unexpected stat failures

In the list of unexpected failures the majority comes from ghci, perf, printer, profiling, rts, safeHaskell and th.

Most probably cause is that the GHC we build is missing some features.

/cc: @bgamari @nomeata

@nomeata

This comment has been minimized.

nomeata commented Feb 9, 2017

Can you put the full log of the testsuite somewhere?

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Feb 9, 2017

@nomeata I don't have the full log, but I attached the list of all failures to this comment.

validate-res.txt

P.S.: This particular log was generated in the default build flavour, but the results are pretty similar to the quick build flavour.

@nomeata

This comment has been minimized.

nomeata commented Feb 10, 2017

Ah, this is on windows? How many of these are also on Linux? But without a full build log it is hard to see what’s going on. Maybe some linker error that affects Template Haskell and GHCi?

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Feb 11, 2017

@nomeata Yes, I develop on Windows and test Hadrian on Linux/OSX via CI only.

Perhaps, you could try validating on Linux and share the log here?

@izgzhen

This comment has been minimized.

Collaborator

izgzhen commented Apr 18, 2017

On my Linux machine for commit c158014 (GHC: 65b185d):

SUMMARY for test run started at Tue Apr 18 15:55:57 2017 CST
 0:14:29 spent to go through
    5847 total tests, which gave rise to
   14389 test cases, of which
    8672 were skipped

      52 had missing libraries
    4743 expected passes
     148 expected failures

      24 caused framework failures
       0 caused framework warnings
       2 unexpected passes
     705 unexpected failures
      43 unexpected stat failures

And some log: https://gist.github.com/izgzhen/a7b8bf3bce035432f5d8fd5605136062

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Apr 18, 2017

Thanks @izgzhen! This looks similar to Windows stats above.

@bgamari bgamari modified the milestones: tree-tremble, ghc-quake May 10, 2017

@izgzhen

This comment has been minimized.

Collaborator

izgzhen commented Sep 29, 2017

A more recent result: Note that I found it is important to run validate after a default build, rather than quickest build. The later case will cause more validation failures. The Make-based system will rebuild everything using a validate flavour, which is very different from how we handle it in Hadrian.

SUMMARY for test run started at Fri Sep 29 12:57:55 2017 HKT
 0:14:21 spent to go through
    6106 total tests, which gave rise to
   23999 test cases, of which
   17951 were skipped

      43 had missing libraries
    5820 expected passes
      91 expected failures

       0 caused framework failures
       0 caused framework warnings
       0 unexpected passes
      87 unexpected failures
       7 unexpected stat failures

Unexpected failures:
   annotations/should_compile/th/annth_make.run       annth_make [bad stdout] (normal)
   annotations/should_compile/th/annth_compunits.run  annth_compunits [bad stdout] (normal)
   codeGen/should_run/T10414.run                      T10414 [exit code non-0] (threaded2)
   concurrent/should_run/conc069.run                  conc069 [bad exit code] (ghci)
   concurrent/should_run/conc069a.run                 conc069a [bad exit code] (ghci)
   concurrent/should_run/conc070.run                  conc070 [bad stdout] (ghci)
   concurrent/should_run/conc072.run                  conc072 [exit code non-0] (threaded2)
   concurrent/should_run/numsparks001.run             numsparks001 [exit code non-0] (threaded1)
   concurrent/should_run/T3429.run                    T3429 [exit code non-0] (debug_numa)
   concurrent/should_run/T5558.run                    T5558 [exit code non-0] (threaded2)
   driver/dynHelloWorld.run                           dynHelloWorld [exit code non-0] (dyn)
   dynlibs/T5373.run                                  T5373 [bad exit code] (normal)
   dynlibs/T3807.run                                  T3807 [bad exit code] (normal)
   dynlibs/T13702.run                                 T13702 [bad exit code] (normal)
   ffi/should_run/ffi014.run                          ffi014 [exit code non-0] (threaded1)
   ffi/should_run/Capi_Ctype_001.run                  Capi_Ctype_001 [bad exit code] (normal)
   ffi/should_run/Capi_Ctype_002.run                  Capi_Ctype_002 [bad exit code] (normal)
   hsc2hs/hsc2hs001.run                               hsc2hs001 [bad exit code] (normal)
   hsc2hs/hsc2hs002.run                               hsc2hs002 [bad exit code] (normal)
   hsc2hs/hsc2hs003.run                               hsc2hs003 [bad exit code] (normal)
   hsc2hs/hsc2hs004.run                               hsc2hs004 [bad exit code] (normal)
   hsc2hs/T3837.run                                   T3837 [bad exit code] (normal)
   hsc2hs/T4340.run                                   T4340 [bad exit code] (normal)
   hsc2hs/T10272.run                                  T10272 [bad exit code] (normal)
   hsc2hs/T12504.run                                  T12504 [bad exit code] (normal)
   plugins/frontend01.run                             frontend01 [bad stdout] (normal)
   pmcheck/should_compile/T11195.run                  T11195 [exit code non-0] (normal)
   printer/T13550.run                                 T13550 [bad stdout] (normal)
   printer/T13942.run                                 T13942 [bad stdout] (normal)
   printer/T13199.run                                 T13199 [bad stdout] (normal)
   profiling/should_run/T2592.run                     T2592 [bad exit code] (profasm)
   profiling/should_run/T3001-2.run                   T3001-2 [exit code non-0] (prof_hb)
   profiling/should_run/scc003.run                    scc003 [bad profile] (profasm)
   profiling/should_run/scc001.run                    scc001 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc002.run                    scc002 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc003.run                    scc003 [bad profile] (prof)
   profiling/should_run/scc003.run                    scc003 [bad exit code] (ghci-ext-prof)
   profiling/should_run/scc005.run                    scc005 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T680.run                      T680 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T2552.run                     T2552 [bad exit code] (ghci-ext-prof)
   profiling/should_run/prof-doc-fib.run              prof-doc-fib [bad exit code] (ghci-ext-prof)
   profiling/should_run/prof-doc-last.run             prof-doc-last [bad exit code] (ghci-ext-prof)
   profiling/should_run/T5559.run                     T5559 [bad exit code] (ghci-ext-prof)
   profiling/should_run/profinline001.run             profinline001 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (profasm)
   profiling/should_run/T5363.run                     T5363 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hc_hb)
   profiling/should_run/T12962.run                    T12962 [bad profile] (profasm)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hb)
   profiling/should_run/T12962.run                    T12962 [bad profile] (prof)
   profiling/should_run/T12962.run                    T12962 [bad exit code] (ghci-ext-prof)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hd)
   quasiquotation/T14028.run                          T14028 [bad stderr] (normal)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hy)
   profiling/should_run/T11627b.run                   T11627b [bad heap profile] (prof_hr)
   rts/bug1010.run                                    bug1010 [exit code non-0] (normal)
   rts/T2047.run                                      T2047 [bad exit code] (normal)
   rts/testwsdeque.run                                testwsdeque [exit code non-0] (threaded1)
   rts/atomicxchg.run                                 atomicxchg [exit code non-0] (threaded1)
   rts/T4850.run                                      T4850 [bad exit code] (normal)
   rts/ffishutdown.run                                ffishutdown [exit code non-0] (threaded1)
   rts/T8209.run                                      T8209 [bad stderr] (ghci)
   rts/T8242.run                                      T8242 [bad stderr] (ghci)
   rts/T9078.run                                      T9078 [exit code non-0] (threaded1)
   rts/T8124.run                                      T8124 [exit code non-0] (threaded1)
   rts/linker_error1.run                              linker_error1 [bad exit code] (normal)
   rts/linker_error2.run                              linker_error2 [bad exit code] (normal)
   rts/linker_error3.run                              linker_error3 [bad exit code] (normal)
   rts/T10728.run                                     T10728 [exit code non-0] (threaded2)
   rts/T10296b.run                                    T10296b [exit code non-0] (threaded2)
   rts/numa001.run                                    numa001 [exit code non-0] (debug_numa)
   rts/T1791/T1791.run                                T1791 [bad exit code] (normal)
   rts/T5644/T5644.run                                T5644 [bad exit code] (optasm)
   simplCore/should_run/simplrun010.run               simplrun010 [bad exit code] (optasm)
   th/TH_spliceE5_prof_ext.run                        TH_spliceE5_prof_ext [bad exit code] (normal)
   th/TH_spliceE5_prof.run                            TH_spliceE5_prof [bad exit code] (normal)
   th/TH_linker/path_with_commas.run                  path_with_commas [bad exit code] (normal)
   typecheck/T13168/T13168.run                        T13168 [bad stderr] (normal)
   typecheck/should_compile/T11462.run                T11462 [exit code non-0] (normal)
   typecheck/should_compile/T11525.run                T11525 [exit code non-0] (normal)
   ../../libraries/base/tests/IO/T12010/T12010.run    T12010 [bad exit code] (threaded1)
   ../../libraries/process/tests/T3231.run            T3231 [exit code non-0] (threaded1)
   ../../libraries/process/tests/T3994.run            T3994 [exit code non-0] (threaded1)
   ../../libraries/stm/tests/stm056.run               stm056 [exit code non-0] (threaded1)
   ../../libraries/unix/tests/forkprocess01.run       forkprocess01 [exit code non-0] (threaded1_ls)
   ../../libraries/unix/tests/fdReadBuf001.run        fdReadBuf001 [bad exit code] (ghci)
   rts/T7815.run                                      T7815 [exit code non-0] (threaded1)

Unexpected stat failures:
   perf/compiler/T1969.run    T1969 [stat not good enough] (normal)
   perf/compiler/T5631.run    T5631 [stat too good] (normal)
   perf/compiler/T5837.run    T5837 [stat too good] (normal)
   perf/compiler/T9872a.run   T9872a [stat too good] (normal)
   perf/compiler/T12150.run   T12150 [stat too good] (optasm)
   perf/should_run/T5205.run  T5205 [stat too good] (normal)
   perf/should_run/T9203.run  T9203 [stat too good] (normal)
@snowleopard

This comment has been minimized.

Owner

snowleopard commented Sep 29, 2017

@izgzhen Thanks for the update! Reducing 705 unexpected failures to 87 unexpected failures simply by switching to the default build flavour is great :)

Indeed, it looks like there is a special flavour for validation.

@bgamari

This comment has been minimized.

Collaborator

bgamari commented Sep 30, 2017

I suppose many of these were due to SplitSections being enabled by default which appears to break the RTS linker. I'm tracking this down in #14291.

@izgzhen

This comment has been minimized.

Collaborator

izgzhen commented Oct 18, 2017

The latest result with default flavour showed a regression:

     165 unexpected failures
       7 unexpected stat failures

I think I shall drop the PR #422 for now, since I still haven't find any clear way to get a better validation result yet.

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 2, 2018

Results generated in ubuntu 17.10 on 2nd March 2017:
steps:

  1. ./build.sh clean

  2. ./build.sh --rebuilding hadrian

  3. ./build.sh validate -VV {I added a third argument to make command in Settings.Builders.Make "VERBOSE=1"}
    Here is the result summary:
    SUMMARY for test run started at Fri Mar 2 20:16:57 2018 IST
    0:19:23 spent to go through
    6262 total tests, which gave rise to
    24521 test cases, of which
    18315 were skipped

    41 had missing libraries
    5945 expected passes
    97 expected failures

    0 caused framework failures
    0 caused framework warnings
    0 unexpected passes
    

    118 unexpected failures
    5 unexpected stat failures

Complete Summary:
results_val2.txt
Error Generated in failures :
errorlog_2.txt

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 2, 2018

Thanks @chitrak7! Hope you'll be able to figure out the cause of some validation failures.

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 2, 2018

@snowleopard 55 of 118 unexpected failures have come from printer/. But I seriously doubt it has something to do with my system configuration and not the build. Can you review their error log once to check this. I have attached the errorlog in the previous comment.

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 2, 2018

@chitrak7 I don't think I can provide any useful comments on the validation log: I haven't looked into validation errors yet, so it's a completely new territory for me.

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 3, 2018

@snowleopard for a lot of failures, the reason is that some executable is not present in inplace. For example,
inplace/bin/check-api-annotations
inplace/bin/check-ppr
inplace/test spaces/runghc

I think solving this might reduce a lot of errors

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 3, 2018

@chitrak7 Aha, interesting! Presumably, these are only needed for the testsuite, so we didn't bother to build them before.

We need to add missing packages here:

https://github.com/snowleopard/hadrian/blob/master/src/GHC.hs#L45-L100

And also need them in the validate rule before running the testsuite. Would you like to give this a try?

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 3, 2018

Yeah, would be a nice start. Would need some time and a lot of help though 😅

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 3, 2018

@chitrak7 Awesome, thanks. Happy to help!

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 5, 2018

Results after adding missing executables:
SUMMARY for test run started at Mon Mar 5 22:30:39 2018 IST
0:20:47 spent to go through
6262 total tests, which gave rise to
24521 test cases, of which
18315 were skipped

  41 had missing libraries
6020 expected passes
  97 expected failures

   3 caused framework failures
   0 caused framework warnings
   0 unexpected passes
  45 unexpected failures
   3 unexpected stat failures

errorlog_3.txt
results_val3.txt
@snowleopard

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 6, 2018

@chitrak7 That's a very promising result! Can you prepare a PR?

Furthermore, it looks like the remaining 45 unexpected failures are due to 41 had missing libraries, which probably means Hadrian doesn't build some libraries needed for the testsuite.

@chitrak7

This comment has been minimized.

Contributor

chitrak7 commented Mar 6, 2018

@snowleopard Currently I am trying to single out more libraries that are causing error. I have identified a couple more. I will try to bring the errors down to 20s and then send the PR. Need a couple more days.

@snowleopard

This comment has been minimized.

Owner

snowleopard commented Mar 6, 2018

@chitrak7 No need to send just one perfect PR. In fact, sending a PR to only adds missing executables would be better, because it's easier to review and understand in isolation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment