CI failure: can't find include file rts/fs.h

[izgzhen]

| Copy file (untracked): _build/stage1/libffi/build/inst/lib/libffi.a => _build/stage1/rts/build/libCffi_thr_l.a
| Successfully built custom library 'libffi'
Installing library in /home/travis/build/snowleopard/hadrian/ghc/_build/stage1/lib/../lib/x86_64-linux-ghc-8.5.20180402/rts-1.0
hadrian: can't find include file rts/fs.h
shakeArgsWith     0.000s    0%                           
Function shake   14.669s    1%                           
Database read     0.000s    0%                           
With database     0.001s    0%                           
Running rules  1314.416s   98%  ======================== 
Total          1329.086s  100%                           
Error when running Shake build system:
* _build/stage1/lib/package.conf.d/rts-1.0.conf
ExitFailure 1

https://travis-ci.org/snowleopard/hadrian/jobs/361477751

https://travis-ci.org/snowleopard/hadrian/jobs/361477747

please leave a word if you are looking into this bug, thanks!

[snowleopard]

I haven't yet looked at this but I have a feeling that this is caused by a missing dependency on fs.h.

These files are copied since #533.

Also, this code may be relevant:

hadrian/src/Rules/Compile.hs

Lines 38 to 58 in 3a68f11

	-- | Discover dependencies of a given source file by iteratively calling @gcc@
	-- in the @-MM -MG@ mode and building generated dependencies if they are missing
	-- until reaching a fixed point.
	needDependencies :: Context -> FilePath -> FilePath -> Action ()
	needDependencies context@Context {..} src depFile = discover
	where
	discover = do
	build $ target context (Cc FindCDependencies stage) [src] [depFile]
	deps <- parseFile depFile
	-- Generated dependencies, if not yet built, will not be found and hence
	-- will be referred to simply by their file names.
	let notFound = filter (\file -> file == takeFileName file) deps
	-- We find the full paths to generated dependencies, so we can request
	-- to build them by calling 'need'.
	todo <- catMaybes <$> mapM (fullPathIfGenerated context) notFound
	if null todo
	then need deps -- The list of dependencies is final, need all
	else do
	need todo -- Build newly discovered generated dependencies
	discover -- Continue the discovery process

Or not :)

[izgzhen]

rts/fs.h is found in ["rts/.","rts/build","rts/../includes","rts/includes","rts/includes/dist-derivedconstants/header","/home/zhen/repos/ghc/_build/stage1/rts/build/.","/home/zhen/repos/ghc/_build/stage1/rts/build/build","/home/zhen/repos/ghc/_build/stage1/rts/build/../includes","/home/zhen/repos/ghc/_build/stage1/rts/build/includes","/home/zhen/repos/ghc/_build/stage1/rts/build/includes/dist-derivedconstants/header"].

The bug is the rts/ prefix.

[izgzhen]

HACK: mkdir rts/rts; cp rts/fs* rts/rts

[izgzhen]

failed with another (probably unrelated) bug: #559

[izgzhen]

@Mistuke @snowleopard any idea on how to understand the HACK and fit it in our code?

[Mistuke]

I don't quite understand why those includes are treated differently than the others. They're copied in tree way before any dependency analysis is done. But we probably don't need to install them anyway. Try in the GHC checkout removing https://github.com/ghc/ghc/blob/master/rts/rts.cabal.in#L132 and the next line.

Note that it's not the building that fails, it's the package installing phase.

I don't have a Linux machine handy atm. but if you can't modify the tree easily I'll go set up a vm.

[snowleopard]

I am with @Mistuke here: I don't quite understand why fs headers are causing issues.

But I'm getting more convinced that I'm right in suspecting that needDependencies is key here. These headers are different from others because they are generated and needDependencies might be unable to discover them.

I'll try to have a look at this issue tonight.

[izgzhen]

@Mistuke

Try in the GHC checkout removing https://github.com/ghc/ghc/blob/master/rts/rts.cabal.in#L132 and the next line.

I tried and it worked.

[snowleopard]

I've added a workaround to fix the fs*.h issue: #566:

    -- TODO: Get rid of this workaround.
    -- See https://github.com/snowleopard/hadrian/issues/554
    root -/- buildDir rtsContext -/- "rts/fs.h"     <~ return "rts"
    root -/- buildDir rtsContext -/- "rts/fs_rts.h" <~ return "rts"

The right way to fix this could be to add fs*.h files to includes/rts which is scanned by Cabal copy. This is how other header files are found.

@izgzhen @Mistuke What do you think?

In the meanwhile, the current workaround works, so I'm lowering the priority of this issue.

[izgzhen]

looks good!

[Mistuke]

Yes that's good, though looking at it again. I probably don't care about having these installed. As the user can't use them easily anyway since they need to know the right name space. I will commit a change removing these from install once I get home. Sorry for the extra work.

…

On Thu, Apr 12, 2018, 03:02 Zhen Zhang ***@***.***> wrote: looks good! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#554 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABH3Kc_4WBicbhLMrGbBaccFGafuR9akks5tnrWhgaJpZM4TExys> .

[izgzhen]

shakeArgsWith    0.000s    0%                           
Function shake   8.356s    1%                           
Database read    0.000s    0%                           
With database    0.000s    0%                           
Running rules  566.332s   98%  =========================
Total          574.688s  100%                           
Error when running Shake build system:
* _build/stage1/lib/package.conf.d/rts-1.0.conf
* _build/stage1/rts/build/rts/fs_rts.h
* rts/fs_rts.h
Error, file does not exist and no rule available:
  rts/fs_rts.h
CallStack (from HasCallStack):
  error, called at src/Development/Shake/Internal/Rules/File.hs:180:58 in shake-0.16.3-D18CK3931i8GsNl2vi6IKs:Development.Shake.Internal.Rules.File

I still saw them in error msg on my machine building the latest commit though :( no yet investigating into it

[Mistuke]

Commit ghc/ghc@111556f removes these files from install-includes

[Mistuke]

Actually I've had to revert this, now I remember why they were there to begin with. The reach-ability analysis forces them to be. Because they're reachable from an exported header. This is all very annoying.

The exported headers are not useful. In fact you can't even call them so moving them to include would be a mistake I think. I think I know a fix that would work both for the testsuite and make/hadrian. I'll whip it up after work today.

[Mistuke]

sigh well that's annoying. the CI failure was unrelated. I put up https://phabricator.haskell.org/D4591 for review to run it through the mac osx builder. if it's green I'll push the removal again.

Sorry for the delay.

[Mistuke]

ghc/ghc@5417c68 done. the CI was clean so the failure was unrelated. You should be able to remove the work-around @snowleopard

[snowleopard]

@Mistuke Great! I'll remove my workaround and will close this today.

[snowleopard]

Done in the above commit. I think we can close this now.