Cannot install pango with new-build when using local copy of gtk2hs-buildtools

[RyanGlScott]

I'm experiencing this issue with both:

$ cabal --version
cabal-install version 2.0.0.1
compiled using version 2.0.1.1 of the Cabal library 
$ cabal-head --version
cabal-install version 2.1.0.0
compiled using version 2.1.0.0 of the Cabal library

To trigger the issue:

$ cabal get pango-0.13.4.0
Unpacking to pango-0.13.4.0/
$ cd pango-0.13.4.0/
$ cabal get gtk2hs-buildtools-0.13.3.1
Unpacking to gtk2hs-buildtools-0.13.3.1/
$ cat > cabal.project << EOF
> packages: .
>           ./gtk2hs-buildtools-0.13.3.1
> EOF
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - gtk2hs-buildtools-0.13.3.1 (lib) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsTypeGen) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsHookGenerator) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsC2hs) (first run)
 - glib-0.13.5.0 (lib:glib) (requires build)
 - cairo-0.13.4.2 (lib:cairo) (requires build)
 - pango-0.13.4.0 (lib:pango) (first run)
Configuring library for gtk2hs-buildtools-0.13.3.1..
Preprocessing library for gtk2hs-buildtools-0.13.3.1..
<build output elided>
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/Main.o )

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:51:18: warning: [-Wdeprecations]
    In the use of ‘autogenModulesDir’
    (imported from Distribution.Simple.BuildPaths):
    Deprecated: "If you can, use 'autogenComponentModulesDir' instead, but if you really wanted package-global generated modules, use 'autogenPackageModulesDir'.  In Cabal 2.0, we avoid using autogenerated files which apply to all components, because the information you often want in these files, e.g., dependency information, is best specified per component, so that reconfiguring a different component (e.g., enabling tests) doesn't force the entire to be rebuilt.  'autogenPackageModulesDir' still provides a place to put files you want to apply to the entire package, but most users of 'autogenModulesDir' should seriously consider 'autogenComponentModulesDir' if you really wanted the module to apply to one component."
   |
51 |     targetDir  = autogenModulesDir lbi
   |                  ^^^^^^^^^^^^^^^^^

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:55:3: warning: [-Wdeprecations]
    In the use of ‘die’ (imported from Distribution.Simple.Utils):
    Deprecated: "Messages thrown with die can't be controlled with Verbosity; use die' instead, or dieNoVerbosity if Verbosity truly is not available"
   |
55 |   die $ "unexpected pango version number: " ++ display version
   |   ^^^
Linking /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup ...
Configuring pango-0.13.4.0...
/u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template: openFile: does not exist (No such file or directory)cabal: Failed to build pango-0.13.4.0. The failure occurred during the
configure step.

Note that this bug will not be triggered if you have installed gtk2hs-buildtools-0.13.3.1 previously using cabal install, as that will cause ~/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template to have been created.

Moreover, this bug does not occur if gtk2hs-buildtools-0.13.3.1 is not put into cabal.project and is instead downloaded from Hackage.

I suspect the issue may have something to do with this line of code in gtk2hs-buildtools-0.13.3.1:

templateFile <- getDataFileName "hierarchyGen/Hierarchy.chs.template"

[RyanGlScott]

I've created https://github.com/RyanGlScott/cabal-gh5164, which minimizes this bug. To use it:

$ git clone https://github.com/RyanGlScott/cabal-gh5164
$ cd cabal-gh5164/
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - setup-lib-0.1 (lib) (first run)
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Configuring library for setup-lib-0.1..
Preprocessing library for setup-lib-0.1..
Building library for setup-lib-0.1..
[1 of 2] Compiling Paths_setup_lib  ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/autogen/Paths_setup_lib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/Paths_setup_lib.o )
[2 of 2] Compiling SetupLib         ( src/SetupLib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/SetupLib.o )
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/Main.o )
Linking /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup ...
setup: /u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1/example.txt: openFile: does not exist (No such file or directory)

[RyanGlScott]

In contrast, here's what happens with cabal install:

$ cabal install setup-lib/
Resolving dependencies...
Configuring setup-lib-0.1...
Building setup-lib-0.1...
Installed setup-lib-0.1
$ cd fails-with-new-build/
$ cabal build
Resolving dependencies...
[1 of 1] Compiling Main             ( dist/setup/setup.hs, dist/setup/Main.o )
Linking ./dist/setup/setup ...
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, dist/build/FailsWithNewBuild.o )

[hvr]

I believe this to be a long-standing issue w/ local packages, because we construct a Paths_ that points to paths that make no sense for nix-style local builds... but I can't seem to find the issue which contains the discussion (and fwiw, a related issue exists for local build-tools #5104).

So the solution would be either to

• have Paths_... contain the actual local filepaths, or
• for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

[RyanGlScott]

Out of curiosity, is there an environment variable that I can set manually to work around this issue?

[hvr]

Sure, it's documented here:

http://cabal.readthedocs.io/en/latest/developing-packages.html?highlight=datadir#accessing-data-files-from-package-code

So you need to set an env-var named normalised_pkgname_datadir (you can cheat by looking at the autogenerated Paths_....hs source code)

And for reference, this here is the place were e.g. new-runsets those env-vars automatically:

cabal/cabal-install/Distribution/Client/CmdRun.hs

Lines 226 to 256 in abaf365

	-- | Construct the environment needed for the data files to work.
	-- This consists of a separate @*_datadir@ variable for each
	-- inplace package in the plan.
	dataDirsEnvironmentForPlan :: ElaboratedInstallPlan
	-> [(String, Maybe FilePath)]
	dataDirsEnvironmentForPlan = catMaybes
	. fmap (foldPlanPackage
	(const Nothing)
	dataDirEnvVarForPackage)
	. toList
	-- | Construct an environment variable that points
	-- the package's datadir to its correct location.
	-- This might be:
	-- * 'Just' the package's source directory plus the data subdirectory
	-- for inplace packages.
	-- * 'Nothing' for packages installed in the store (the path was
	-- already included in the package at install/build time).
	-- * The other cases are not handled yet. See below.
	dataDirEnvVarForPackage :: ElaboratedConfiguredPackage
	-> Maybe (String, Maybe FilePath)
	dataDirEnvVarForPackage pkg =
	case (elabBuildStyle pkg, elabPkgSourceLocation pkg)
	of (BuildAndInstall, _) -> Nothing
	(BuildInplaceOnly, LocalUnpackedPackage path) -> Just
	(pkgPathEnvVar (elabPkgDescription pkg) "datadir",
	Just $ path </> dataDir (elabPkgDescription pkg))
	-- TODO: handle the other cases for PackageLocation.
	-- We will only need this when we add support for
	-- remote/local tarballs.
	(BuildInplaceOnly, _) -> Nothing

[RyanGlScott]

Thanks, that's very helpful.

A related question: does new-build put the data-files somewhere in dist-newstyle? I just searched for example.txt, but couldn't find it anywhere in my dist-newstyle.

[RyanGlScott]

For the time being, I'm finding myself having to hardcode the directory in which the data-files live to make this work. For instance:

$ setup_lib_datadir="$(pwd)/setup-lib" bash -c 'cabal new-build all'
In order, the following will be built (use -v for more details):
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/build/FailsWithNewBuild.o )
Example data file

Example data file

Obviously, this is quite ugly, and won't scale if your data-files are spread out over multiple directories. But since new-build apparently doesn't put data-files into dist-newstyle, I don't see any other way to make this work at the moment.

[hvr]

A related question: does new-build put the data-files somewhere in dist-newstyle?

Maybe for (non-local) "inplace" packages (I'd have to check), but certainly not for local "inplace" ones.

[RyanGlScott]

Note that stack does the correct thing here:

$ stack build fails-with-new-build                   
setup-lib-0.1: configure (lib)
setup-lib-0.1: build (lib)
setup-lib-0.1: copy/register
fails-with-new-build-0.1: configure (lib)
[1 of 2] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/Setup.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/Main.o )
[2 of 2] Compiling StackSetupShim   ( /home/rgscott/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/StackSetupShim.o )
Linking /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/setup ...
Example data file
             
Configuring fails-with-new-build-0.1...
fails-with-new-build-0.1: build (lib)
Example data file
             
Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/FailsWithNewBuild.o )
fails-with-new-build-0.1: copy/register
Example data file
             
Installing library in /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/fails-with-new-build-0.1-BoDEqEiYoIHEwENsULGL10
Example data file
             
Registering library for fails-with-new-build-0.1..
Completed 2 action(s).

$ more ./setup-lib/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/autogen/Paths_setup_lib.hs
{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -fno-warn-missing-import-lists #-}
{-# OPTIONS_GHC -fno-warn-implicit-prelude #-}
module Paths_setup_lib (
    version,
    getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir,
    getDataFileName, getSysconfDir
  ) where

import qualified Control.Exception as Exception
import Data.Version (Version(..))
import System.Environment (getEnv)
import Prelude

#if defined(VERSION_base)

#if MIN_VERSION_base(4,0,0)
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#else
catchIO :: IO a -> (Exception.Exception -> IO a) -> IO a
#endif

#else
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#endif
catchIO = Exception.catch

version :: Version
version = Version [0,1] []
bindir, libdir, dynlibdir, datadir, libexecdir, sysconfdir :: FilePath

bindir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/bin"
libdir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/setup-lib-0.1-8hekJdedMBc9fTEHUxKQOX"
dynlibdir  = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2"
datadir    = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
libexecdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/libexec/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
sysconfdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/etc"

getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir, getSysconfDir :: IO FilePath
getBinDir = catchIO (getEnv "setup_lib_bindir") (\_ -> return bindir)
getLibDir = catchIO (getEnv "setup_lib_libdir") (\_ -> return libdir)
getDynLibDir = catchIO (getEnv "setup_lib_dynlibdir") (\_ -> return dynlibdir)
getDataDir = catchIO (getEnv "setup_lib_datadir") (\_ -> return datadir)
getLibexecDir = catchIO (getEnv "setup_lib_libexecdir") (\_ -> return libexecdir)
getSysconfDir = catchIO (getEnv "setup_lib_sysconfdir") (\_ -> return sysconfdir)

getDataFileName :: FilePath -> IO FilePath
getDataFileName name = do
  dir <- getDataDir
  return (dir ++ "/" ++ name)

[hvr]

@RyanGlScott this looks a lot like what the old cabal sandbox did as well...

[RyanGlScott]

Indeed, I can confirm that this works with sandboxes as well.

This is an annoying enough issue (I experience it several times per day) that I'm inclined to figure out how to fix it. stack fixes this issue by wiring in a different datadir. Is there an equivalent code path for new-build that would accomplish the same thing?

[hvr]

Well, first we'd need to agree which of the two variants mentioned in my comment #5164 (comment) we want to pursue (there's a reason we didn't just go w/ the obsolete cabal sandbox approach, but I don't remember the details); I remember some discussion about this in another ticket, and it would be worth locating that again to learn about the conclusion from there.

PS: related #4120 and #4558 (comment) (but I don't think that's the discussion I was thinking of)

/cc @phadej @23Skidoo

[23Skidoo]

• for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

I'm +1 on this idea, what are the arguments against?

[RyanGlScott]

I'd be in favor of that as well.

[hvr]

@23Skidoo Can't think of any showblocker arguments against either.