Indexing packages coming with GHC

[qrilka]

I was trying to index bytestring and found out that it couldn't be done with haskell-indexer and also it looks like it breaks package db by unregistering bytestring so I needed to run stack setup --reinstall to fix the situation. Also it seems to be related to the question from @chrisdone in #79 about base but it looks like base is not enough and dependencies of GHC which come with it need special treatment. I see in #56 scripts from you @robinp but it looks like it doesn't cover libraries which are in git submodules of GHC repo - why didn't they get indexed? BTW shouldn't those scripts also be included into this repo with some file describing how to index GHC itself (at the moment it's not clear to me where that /opt/ghc/bin/ghc come from - probably it's about your custom install of GHC into /opt/ghc?)

[robinp]

Hey, thanks for the feedback. To tell the truth, GHC indexing was quite hacky (substituting the ghc calls from the make log), there's certainly room for improvement.

As for indexing basic libs such as base mentioned by @chrisdone or bytestring, indeed, I tried with stack but due to the forced unregister (which was to try to force stack to recompile, with varying success) that doesn't play nicely with your install, and on top of that doesn't even work.

I misremembered, it was cabal new-build, not stack which I used to build base and others, see haskell/cabal#4209 for the --install-base-libraries flag to cabal. Unfortunately I don't have out of the box wrappers for cabal new-build as there is for stack.

I agree, GHC indexing scripts would be nice to ship too. In 2 months I'll have a bit more bandwidth, until then I'm constrained to reviewing PRs.

[qrilka]

@robinp if you could give some hints I could help you implementing that, good source navigation will help in our projects and for Haskell community as well

[mpickering]

Now I have implemented the plugin I tried to get it to integrate it into hadrian so that I could use it in GHC's build process. To do this I had to make hadrian support building a 3rd GHC stage so that plugins could be run on base but then I ran into problems writing a hadrian rule to install a package from hackage as I couldn't get it to register with a consistent name in the package database.

[robinp]

I revisited this, state seems to be that with plain userland cabal or stack indexing base is quite impossible due to all kind of safeguards.

@mpickering mentions using the source plugin with the GHC build system above, which sounds adventourous.

In the mean time I think one could just hijack the compilation commands from a GHC build - when doing make, all the compiler invocations are helpfully logged to the console. We are interested in the stage1 compiler compiling the base packages and maybe the stage2 compiler itself, so one could replay the inplace/compiler/stage1-ghc invocations to ghc_kythe_wrapper. The resulting entry set could be injected into the indexing pipeline (well, with open-source kythe tools it is just an other entries file).

I'll have a look at this, and refine the method a bit. /cc @jinwoo @judah