Add alex to build-tool-depends remove separate code gen

[andreabedini]

In Distribution.Fields.Lexer we have this note.

-- [Note: bootstrapping parsec parser]
--
-- We manually produce the `Lexer.hs` file from `boot/Lexer.x` (make lexer)
-- because bootstrapping cabal-install would be otherwise tricky.
-- Alex is (atm) tricky package to build, cabal-install has some magic
-- to move bundled generated files in place, so rather we don't depend
-- on it before we can build it ourselves.
-- Therefore there is one thing less to worry in bootstrap.sh, which is a win.
--
-- See also https://github.com/haskell/cabal/issues/4633

The issue linked in the note mentions that alex has build-type: Simple since Feb 2018. Moreover, bootstrap.sh, has been replaced by a Python script that seems to be more powerful.

I haven't actually tried to bootstrap yet, so opening this as a draft.

---

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.
• Include manual QA notes if your PR relates to cabal-install.

Bonus points for added automated tests!

[andreabedini]

The most annoying part in testing this is that cabal-bootstrap-gen doesn't know about xdg-dirs :-/

cp dist-newstyle/cache/plan.json bootstrap/linux-8.10.7.plan.json
cd bootstrap && cabal v2-run -v0 cabal-bootstrap-gen -- linux-8.10.7.plan.json \
	| python3 -m json.tool > linux-8.10.7.json
cabal-bootstrap-gen: /home/andrea/.cabal/config: withBinaryFile: does not exist (No such file or directory)
Expecting value: line 1 column 1 (char 0)
make: *** [Makefile:195: bootstrap-json-8.10.7] Error 1

[andreabedini]

I am afraid this need a fair bit of rework. It's not only bootstrap.py that needs to change but also cabal-bootstrap-gen. Its view of the plan is too simplistic and does not understand per-component builds.

I notice that cabal-plan already gives us a good "execution plan":

❯ cabal-plan topo --plan-json linux-8.10.7.plan.json --show-flags
cabal-install-3.11.0.0 exe:cabal +lukko +native-dns
cabal-install-3.11.0.0 +lukko +native-dns
safe-exceptions-0.1.7.3
resolv-0.1.2.0
regex-posix-0.96.0.1 -_regex-posix-clib
regex-base-0.94.0.2
hackage-security-0.6.2.3 +base48 +cabal-syntax +lukko -mtl21 -old-directory +use-network-uri
zlib-0.6.3.0 -bundled-c-zlib -non-blocking-ffi -pkg-config

This does include the components to build in the correct order and also exe-depends like alex.

Edit: cabal-bootstrap-gen already uses cabal-plan, there's hope!

[andreabedini]

This seems to work now. I was surprised we have no alex-options in .cabal.

I could pass --latin1 as an in-file annotation but I am not sure if I can do the same with --ghc.

The build script now performs a per-component build which works in this case but won't work in full generality (because it doesn't respects qualified goals and flattens everything different goals in a single plan).

[andreabedini]

D:\a\cabal\cabal\cabal-testsuite\PackageTests\MultiRepl\EnabledSucc\cabal.dist\work\.\dist\multi-out-5488\paths\pkg-b-0-inplace: removeDirectoryRecursive:removeContentsRecursive:removePathRecursive:removeContentsRecursive:removePathRecursive:DeleteFile "\\\\?\\D:\\a\\cabal\\cabal\\cabal-testsuite\\PackageTests\\MultiRepl\\EnabledSucc\\cabal.dist\\work\\dist\\multi-out-5488\\paths\\pkg-b-0-inplace": permission denied (The process cannot access the file because it is being used by another process.)

CI has some hiccups. I'll kick the tire again.

[ulysses4ever]

I'm sorry i need some time to look into it: I'm not terribly familiar with how we use alex and bootstrapping in general. I'd be happy to review, but I'll need to find a chunk of time to first educate myself in these matters a bit.

[andreabedini]

@ulysses4ever No worries. Nothing urgent about this ☺️

[ulysses4ever]

@andreasabel we're lacking experience here. Could you maybe provide a review for this alex-related PR?

[andreasabel]

From afar, LGTM, provided it works. Thanks!
The --ghc flag to alex should be supplied automatically by cabal, so I think we need not worry about that.
There are unnecessary diffs in the linux-*.json file due to reorderings. Maybe the generator for these files could be fixed so that random reorderings do not happen.

[andreasabel]

Minor comment: Would it make sense to order this list alphabetically by package name so that diffs are more stable.
In this case, the diff is just a reordering, this could/should be avoided.

[andreabedini]

That's a good idea 👍

[andreabedini]

Actually, no 😂

The lines you highlight are from the builtin section (aka pre-installed packages). Those don't really need to follow a specific order, and they are used only for validation in bootstrap.py. But the other list of packages has to be in topological order otherwise we will find missing dependencies. This is why it's a list and not a dictionary.

I can turn the builtin section into a dict, but I'd avoid changing the rest.

[andreasabel]

From the alex side, this LGTM.
(I am not qualified on the bootstrap process here, so I cannot review this part sensibly.)

[andreabedini]

Do we know who is using this script? Can we get their input?

[andreabedini]

@arrowd @maralorn Do you use this bootstrap script?

[arrowd]

Yes, I do. The change seems fine to me, but I only had a quick look.

[maralorn]

I have no clue. Maybe @sternenseemann knows better.
The only thing I can say is that we in principle build happy and alex for bootstraping ghc, so they should be available when building Cabal as well.

[sternenseemann]

We don't use cabal-install anywhere in package builds (except for ghcjs which is only built when cabal-install is easily attainable), so cabal-install is nothing we need to bootstrap at all, thanks to Setup.hs and GHC taking care of bootstrapping Cabal.

[sternenseemann]

Alpine Linux definitely uses it, so maybe @nmeum could take a look?

[andreabedini]

Can I haz another review?

[andreabedini]

That was quick @Kleidukos! <3

[Bodigrim]

(Upd.: Whoops, sorry, Lexer.x was removed in #9198, not here. Moving the bulk of the comment there)

When evaluating changes related to bootstrapping, it might help to bump Cabal submodule in GHC source tree and estimate the impact. There are test-bootstrap jobs: if they pass fine, we are all good.