Rewrite to use SQL backend

[igfoo]

Consensus currently seems to be that Hackage 2 should use an SQL database rather than acid-state.

See thread: http://www.haskell.org/pipermail/cabal-devel/2012-September/009069.html

[dcoutts]

Not a blocker for hackage 2 release.

[dag]

It's slow. It takes about 5 minutes for me to stop and then start the
server. It's actually surprising just how slow it is, so it might be
possible/easy to get this down to seconds, but it still won't be
instantaneous.

If you use the remote backend and keep an acidServer running, it will be instantaneous. Of course you'll need to restart the acidServer in some situations, like if you implement a new event, but at least it won't be all the time.

Only a single process can use the database at once. For example, if
the admins want a tool that will make it easier for them to approve
user requests, then that tool needs to be integrated into the Hackage
server (or talk to it over HTTP), rather than being standalone.

Same solution, use the remote backend.

The database is relatively opaque. While in principle tools could be
written for browsing, modifying or querying it, currently none exist
(as far as I know).

Also remote backend. Then you can write a $PWD/.ghci to import the right stuff and connect to the remote acidServer. Also useful is adding a Query event that's just ask, then you can inspect the state in ghci and it's not so opaque anymore.

The above 2 points mean that, for example, there was no easy way for
me to find out how many packages use each top-level module hierarchy
(Data, Control, etc). This would have been a simple SQL query if the
data had been in a traditional database, but as it was I had to write
a Haskell program to process all the package .tar.gz's and parse the
.cabal files manually.

In the setup I suggested, you'd just enter ghci, grab the full state and mung it with Haskell like you would any Haskell value. One problem though, if the state is as big as you say I think it would need to be copied, over the socket, to the ghci process in full. So yea it's not perfect.

Also, not sure if you're using ixset for hackage2? If not, it might've helped in that case because you could have arbitrary indexes like with an SQL database, including indexing on exposed-modules. However, ixset has some problems and it's going to be phased out in favor of a new "data-store" package by Palmik, so not sure this is a good time to invest time in porting a large code base to ixset ...

[dcoutts]

So far it's all going fine with the acid-state stuff, no current plans to change.