Automatic hoogle database for installed packages #395

Open
bos opened this Issue May 24, 2012 · 11 comments

Projects

None yet

7 participants

@bos
Contributor
bos commented May 24, 2012

(Imported from Trac #402, reported by @peaker on 2008-11-11)

The hoogle package is very useful to search the standard library. Unfortunately, it does not search packages that you have installed which are not part of the stdlib.

It would be great if every package installed always had its haddock generated, and from its .txt file a .hoo file was generated (e.g: hoogle --convert), and then all .hoo files sat in some centralized directory which hoogle would then use for its searches.

This would be great documentation, and let you hoogle for anything that you can import.

@bos
Contributor
bos commented May 24, 2012

(Imported comment by @peaker on 2008-11-11)

Using hoogle --convert on the .txt files directly will not work until this bug is fixed:
http://hackage.haskell.org/trac/ghc/ticket/2766

@bos
Contributor
bos commented May 24, 2012

(Imported comment by guest on 2008-11-11)

#3766 has been fixed.

More importantly, I'm not sure it was a problem. I'm still on 6.10.1, but my Hoogle had no problem converting an xmonad-contrib.txt and searching the .hoo file.

@bos
Contributor
bos commented May 24, 2012

(Imported comment by guest on 2008-12-17)

#3766 would only have been a problem for certain libraries, so it won't effect most things.

This would be great if someone could do it :-) I'll do any Hoogle changes that are necessary, but don't have time to do the Cabal side.

@bos
Contributor
bos commented May 24, 2012

(Imported comment by guest on 2008-12-17)

I don't think many Hoogle changes are necessary, but it's unclear how it's going to be handled. Is Cabal going to dump .hoo files into Hoogle's data-directory, or should Hoogle be going and looking elsewhere for .hoo files or what?

Personally, I like the idea of Cabal dumping the generated .hoo file into Cabal's data-directory myself. Simple.

But that opens other questions. As ndm says, 'Hoogle could
look in the same directory as haddock does for its documentation
databases, Hoogle could dump its path to stdout, hoogle could dump the converted files to its path'.

Here again I favor the simple solution of dumping the path to stdout, but another possibility is exposing Hoogle's library and thus Paths_hoogle (I think that'd work).

Finally, there becomes the question of what .hoos Hoogle should look in by default. ndm: "There should certainly be an
all database, and an all libraries on hackage database, and an all
installed libraries database. Which one of those is default is a good question, I'd guess at all installed libraries - but not certain."

@bos
Contributor
bos commented May 24, 2012

(Imported comment by dschoepe on 2008-12-17)

This patch should resolve this issue at least on cabal-install's part:
http://article.gmane.org/gmane.comp.lang.haskell.cabal.devel/6566

The question of what is a good default for hoogle still remains though.

@bos
Contributor
bos commented May 24, 2012

(Imported comment by dschoepe on 2010-06-20)

I've been using my patch for the past few months now and it appears to work fine, so once the required haddock-patch(about generating hoogle and html output in one call[1]) is applied, this patch should be ready to be applied.

[1] http://thread.gmane.org/gmane.comp.lang.haskell.cabal.devel/5617

@bos
Contributor
bos commented May 24, 2012

(Imported comment by @dcoutts on 2010-11-06)

I applied the other patch about generating hoogle and html output in one go. I've looked at this patch but I think it needs a bit more thought.

So, the general idea is to create a binary .hoo file for each package right? Generating a merged binary database would be a separate task, not covered by this patch. There are two issues:

  1. The code in the doc step of cabal-install is assuming a certain layout of the build tree by reading and writing files directly in the build tree. This will not work for packages using build-type custom. It would be ok if we were copying into a temporary image dir before doing the real installation. In that case writing extra files into the temporary image dir would be fine. As it is, I think this feature of generating a per-package .hoo db would be better in the build system itself, in the Cabal lib.
  2. according to the [hoogle manual](http://www.haskell.org/haskellwiki/Hoogle#Database_Creation), the .hoo files of dependencies need to be specified as well.
Since each .txt can be converted into a .hoo, and the final .hoo is not really useful except as something to use to merge into a db that users will actually use, perhaps it would be better to just generate the .txt files per-package and then generate a big merged one. Alternatively, why generate the .txt files if we could just generate the .hoo files?

Is there any point in generating both .txt and .hoo files for each package?

More minor comments about the patch. It should use the existing framework for running programs. This would cut down on the code and integrate it properly with the Cabal exception handling and logging stuff.

@23Skidoo 23Skidoo was assigned Dec 27, 2012
@chrisdone
Member

@23Skidoo

So after surveying the #haskell channel it's clear that we as a community are generally completely bad at searching and getting documentation of our packages. We use hoogle online if it's available, or we google the module name if not, and locally we read docs if we have it generated, or we grep. Really, what we should be able to do is install a package and immediately be able to run hoogle ThatPackage.foo and get (1) the type information, (2) the documentation. This can be used from within GHCi. Additional to that, a hoogle search for all installed packages will trivially allow “automatically import names that I use”, among other things. It's kind of a Big Deal, it seems generally agreed, speaking in terms of productivity, to get this sorted out. So here we go.

@dcoutts

In that case writing extra files into the temporary image dir would be fine. As it is, I think this feature of generating a per-package .hoo db would be better in the build system itself, in the Cabal lib.

We have the ability to generate Hoogle from here already, so do you mean “let's change it from generating .txt files to .hoo files”? If so, that sounds good, I can do that. I already had a read through the codebase of how Haddock is implemented. I was considering implementing another command, hoogle, in the Simple setup, but given that hoogle is dependent upon Haddock (or we implement our own .txt outputter, which I think nobody wants nor would see reason to do) that it's probably reasonable to leave the hoogle generation as a flag of the Haddock command, but that we, in there, run an extra step to perform the conversion from .txt to .hoo, replacing or complementing the .txt file(s). I've seen how Cabal deals with programs, so I think I would add hoogle as a program like that, but ultimately it's a rather simple program with little configuration so it should be a few lines.

That would be step one of this issue.

Step two would be to make cabal-install, much like it does with Haddock, regenerate a database (like Haddock with its index regeneration), or merge, whatever gives us an updated hoogle database in a location specified in a similar way to how the global documentation index file's path is specified (in the config). Also there should be the config option to make this automatic a la (Documentation: True). So ideally the result would give us something like:

foo$ runhaskell Setup.hs haddock --hoogle
…
Generating documentation in: dist/html/foo/foo.html
Generating hoogle database in: dist/hoogle/foo.hoo

And then, due to Hoogle: True (or something like that), this would also work:

$ cabal install foo
Generating documentation in: dist/html/foo/foo.html
Generating hoogle database in: dist/hoogle/foo.hoo
Installing to ~/.cabal/blah/blah…
Re-generating documentation … in ~/.cabal/share/doc/index.html
Re-generating hoogle database … in ~/.cabal/share/hoogle/index.hoo
$ hoogle Foo.bar
Lorem ipsum hadoctum documentas.
Foo.bar :: Mu -> Zot

This should also, if we do it by the book according to Cabal, Just Work™ with cabal-dev and those sandboxing libraries, until Cabal has its own sandboxing released. It is important that documentation and type information reflects what I've installed, locally in a directory, or globally.

Any problems with any of the above?

@nkpart
nkpart commented Nov 27, 2013

This sounds awesome. Is the work described here still applicable?

@m0davis
m0davis commented Apr 27, 2014

I don't think cabal has been updated yet with the aforementioned feature. Until it is, this script may be useful: https://groups.google.com/forum/#!topic/haskell-cafe/K9wba99z_fE

@gilligan

Will this ever be addressed again in some way ?

@m0davis : what is hoogle-install which is used in the script you mentioned. I also wonder how this could be adapted for use with sandboxes.

@ttuegel ttuegel added this to the _|_ milestone Apr 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment