cabal-install-3.8.1.0 update and sdist require ghc in PATH

[phadej]

Otherwise

Project settings changed, reconfiguring...
creating /__w/splitmix/splitmix/dist-newstyle/cache
creating /__w/splitmix/splitmix/dist-newstyle
creating /__w/splitmix/splitmix/dist-newstyle/cache
Compiler settings changed, reconfiguring...
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:370:14 in Cabal-3.8.1.0-inplace:Distribution.Simple.Utils
Error: cabal-3.8.1.0: The program 'ghc' version >=7.0.1 is required but it

I guess I'm required to pass --with-compiler now?

[phadej]

Error: cabal-3.8.1.0: The program 'ghc' version >=7.0.1 is required but it could not be found.

cabal sdist is not runnable without GHC in PATH, and cabal sdist doesn't have --with-compiler option.

This is major blocker for using cabal-install-3.8.1.0.

[Mikolaj]

I don't know when it changed, but my guess would be that cabal sdist now requires GHC for some of its checks. @ffaf1: do you have any inkling?

Would enabling --with-compiler be a sufficient fix?

[phadej]

Would enabling --with-compiler be a sufficient fix?

Sufficient but ugly. Even if cabal.project supported conditionals based on compiler version, sdist could still work without knowing anything about a compiler, taking union of all possibly available packages. sdist for single package doesn't need a compiler.

[RyanGlScott]

Indeed, this prevents cabal-install-3.8.1.0 from being usable from within haskell-ci, which avoids putting compilers on the PATH by design.

[Mikolaj]

Uh, oh, cabal update needing GHC sounds definitely like a bug.

[Mikolaj]

That's a probably a release blocker. Too bad, we've already released. Are there any workarounds for haskell-ci?

A pity nobody tried 3.8.1.0-rc1 with haskell-ci. Is that reasonably easy to do for a haskell-ci user, for next time (3.8.1.0-rc1 was installable with ghcup)?

[phadej]

A pity nobody tried 3.8.1.0-rc1 with haskell-ci. Is that reasonably easy to do for a haskell-ci user, for next time (3.8.1.0-rc1 was installable with ghcup)?

I wasn't aware it was available through ghcup. It shouldn't be hard to test release candidates in the future. Though there is little insensitive as older cabal-install seems to work well enough. And that is a workaround for now: stick to using older cabal-install.

[ffaf1]

Maybe 32259a1?

cabal/cabal-install/src/Distribution/Client/ProjectPlanning.hs

Line 336 in 32259a1

(compiler, Platform arch os, _) <- configureCompiler verbosity distDirLayout ((fst $ PD.ignoreConditions projectConfigSkeleton) <> cliConfig)

and

cabal/cabal-install/src/Distribution/Client/ProjectPlanning.hs

Lines 419 to 423 in 32259a1

	liftIO $ info verbosity "Compiler settings changed, reconfiguring..."
	result@(_, _, progdb') <- liftIO $
	Cabal.configCompilerEx
	hcFlavor hcPath hcPkg
	progdb verbosity

Follow the trail and you will end up with the dreaded requireProgramVersion.

[Mikolaj]

But is this (rebuildProjectConfig) really called even in cabal update?

[ffaf1]

Yes, see ProjectOrchestration.hs

cabal/cabal-install/src/Distribution/Client/ProjectOrchestration.hs

Lines 218 to 234 in 32259a1

	establishProjectBaseContextWithRoot
	:: Verbosity
	-> ProjectConfig
	-> ProjectRoot
	-> CurrentCommand
	-> IO ProjectBaseContext
	establishProjectBaseContextWithRoot verbosity cliConfig projectRoot currentCommand = do
	cabalDir <- getCabalDir
	let distDirLayout = defaultDistDirLayout projectRoot mdistDirectory
	httpTransport <- configureTransport verbosity
	(fromNubList . projectConfigProgPathExtra $ projectConfigShared cliConfig)
	(flagToMaybe . projectConfigHttpTransport $ projectConfigBuildOnly cliConfig)
	(projectConfig, localPackages) <-
	rebuildProjectConfig verbosity

and

cabal/cabal-install/src/Distribution/Client/ProjectOrchestration.hs

Lines 205 to 212 in 32259a1

	establishProjectBaseContext
	:: Verbosity
	-> ProjectConfig
	-> CurrentCommand
	-> IO ProjectBaseContext
	establishProjectBaseContext verbosity cliConfig currentCommand = do
	projectRoot <- either throwIO return =<< findProjectRoot Nothing mprojectFile
	establishProjectBaseContextWithRoot verbosity cliConfig projectRoot currentCommand

and then CmdUpdate.hs

cabal/cabal-install/src/Distribution/Client/CmdUpdate.hs

Lines 111 to 116 in 32259a1

	updateAction :: NixStyleFlags () -> [String] -> GlobalFlags -> IO ()
	updateAction flags@NixStyleFlags {..} extraArgs globalFlags = do
	let ignoreProject = flagIgnoreProject projectFlags
	projectConfig <- withProjectOrGlobalConfig verbosity ignoreProject globalConfigFlag
	(projectConfig <$> establishProjectBaseContext verbosity cliConfig OtherCommand)

and up to the original cabal update call.

[ulysses4ever]

Is it about the time to ping @gbaz?..

[phadej]

I'd suspect changes related to --ignore-project implementation, not conditionals.

So possibly: #8109, but for be sure you shouldgit bisect.

[ffaf1]

git bisect agrees with my diagnosis: " 32259a1 is the first bad commit "

#7783

[gbaz]

Good diagnosis! Looking now at the logic there, I'm not sure if this change in behavior is wrong, though it is surprising.

In particular, that PR allows conditionals on ghc versions to be in project files. And... project files I believe have options that may affect even update and sdist? So If one does not have a ghc version, then when one reads a project file, as update and sdist must, then what should happen when a conditional is encountered?

A change would I think have to be some sort of "defaulting" in the specific cases of commands that don't strictly need ghc, which seems pretty strange.

It might be better to just change haskell-ci scripts to always make sure a ghc made available even for these commands?

[phadej]

It might be better to just change haskell-ci scripts to always make sure a ghc made available even for these commands?

That is lazy approach (EDIT: and even then sdist command should have --with-compiler. haskell-ci wont put ghc in $PATH). Why can't you just fix cabal?

E.g. when haskell-ci calls cabal update there is no project file. What does cabal needs GHC then for. Why it spends cycles looking for it?

[phadej]

And to expand on haskell-ci philosophy. Because haskell-ci doesn't put GHC in path, we found numerous bugs in cabal where it was looking for ghc and not using --with-compiler provided one. Also bugs in packages which expected ghc (or some other stuff) to be in PATH.

CI is a good place to work in highly hygienic environment, without humongous implicit state.

I'd appreciate if cabal supported such workflows in the future.

[gbaz]

"when haskell-ci calls cabal update there is no project file. What does cabal needs GHC then for. Why it spends cycles looking for it"

As I described, update and sdist may both be affected by options in project files. Project files may now have conditionals. Those conditionals may branch on version of ghc (or an arch). Knowing how to branch on a version of ghc or its supported arch requires finding the ghc available and querying its version.

A potential refactor would be to only query for ghc when such conditionals exist, but that would be extremely ad-hoc and also not at all fit with the current pipeline in terms of when things are pure and when they are not.

Your argument for not having a systemwide ghc but only using --with-compiler makes a lot of sense, as a way to shake out bugs. But that said, it seems like changing the haskell ci scripts to pass those flags to these two commands seems straightforward?

(edit: that said, its certainly an issue that sdist doesn't have a --with-compiler option, and I do think that would be an important fix, regardless)

[phadej]

A potential refactor would be to only query for ghc when such conditionals exist, but that would be extremely ad-hoc and also not at all fit with the current pipeline in terms of when things are pure and when they are not.

It won't be ad-hoc. There is ProgramDb in Cabal was designed that you would configure a program at most once. The threading is just not applied through the cabal-install.

[gbaz]

What I mean is the following -- if we made that change, then using it, haskell-ci would work on most packages. But there would eventually be a package that had a conditional branch in its cabal.project, and haskell-ci would fail to work on that project, because in that specific instance it would need to figure out which ghc to use in order to fully parse the project.

So I think it is better to make a change that lets everything always work on all projects, and that change is extending sdist to have a --with-compiler flag.

[phadej]

haskell-ci would work on most packages. But there would eventually be a package that had a conditional branch in its cabal.project,

haskell-ci always generates own cabal.project files, it doesn't use projects' own ones.

[phadej]

... and as I said, even then taking an union of all branches should be enough for sdist. It doesn't need to know compiler.

[jneira]

... and as I said, even then taking an union of all branches should be enough for sdist. It doesn't need to know compiler.

hmm if you have

if impl(ghc >= x.y.z)
  packages: ./package1
else
  packages: ./package1 ./package2

cabal sdist all would not behave the same way having in account ghc versions or fusing all branches

EDIT: for cabal update the counterexample would be even clearer:

if impl(ghc >= x.y.z)
  active-repositories: repo1
else
  active-repositories: repo1, repo2

[fgaz]

What about requiring ghc only if there are impl(ghc) branches?

[jneira]

cabal sdist --ignore-project should not query ghc neither and it fails with no ghc in PATH

[jneira]

haskell-ci always generates own cabal.project files, it doesn't use projects' own ones.

And there are no plans to generate cabal.project files with conditionals? at a first look it could be useful for haskell-ci.
To have even the chance to use them haskell-ci, the command would need -w and no remove the need for ghc (so remove the use of ghc conditionals)

[phadej]

cabal sdist all would not behave the same way having in account ghc versions or fusing all branches

IMO it should not account for ghc ever. cabal sdist all should sdist all packages, regardless of which ghc happens to be in $PATH. Same applies for cabal update, updating more repositories won't destroy anything.

[roberth]

IMO it should not account for ghc ever.

This is not just an opinion. cabal sdist is used for uploading to hackage, which means that the sdists must include the source files for all supported configurations, regardless of conditionals, including conditionals that involve ghc.

[jneira]

Hmm sdist is not used to upload to hackage diretly, you could use another tool to generate the tarball to be uploaded and you can use the tar generated by sdist for other purposes.
Otoh you dont upload cabal.project to hackage so changing manually the field packages in a cabal.project.local before a sdist would be equivalent to use one or another ghc in my example (or, he, upload only some of the tar generated or whatever) and hackage will never know why there are those packages and no other ones.

[roberth]

Hmm sdist is not used to upload to hackage diretly

My release scripts do call sdist before uploading. IIRC cabal upload fails if you don't call cabal sdist first. If there's a better way to do it, I'm all ears.

Regardless, the current behavior is a breaking change.

Otoh you dont upload cabal.project to hackage so changing manually the field packages

Are you suggesting that the ghc dependency is only due to the cabal.project feature?
My first encounter with this bug was in a Nixpkgs feature that does not use cabal.project, so that was a little confusing. My release scripts don't rely on cabal.project either, so I guess I'm mostly unaffected then.

Shouldn't release commands always produce the same behavior regardless of the Haskell and OS platforms they run on?

[Mikolaj]

Also affects GHC CI: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/8943

[gbaz]

resolved by #8358